Static files served without recording hit to access_log

Overview

Static files (index.html) can be served without logging a request in access_log.

Cause

This occurs when mod_pagespeed is enabled. Pagespeed intercepts the request higher in Apache’s processing axis before mod_log_config and serves the static file from its optimized cache if present. This does not affect files which have a Pragma: no-cache header such as with PHP files.

Solution

Disable mod_pagespeed. If visitor statistics are necessary, then consider utilizing Google Analytics which is accessible from within apnscp.

Leave a Reply

Working with HTTP rate-limiting

Overview

All HTTP servers enforce a collection of HTTP rate-limiting to reduce abuse and achieve a high reliability. This system is built on a fork of mod_evasive, which implements an interval-based bean counter, in other words it begins counting URI requests for a given duration once the first request is received.

There are two classes of URI requests, pages in total and same page requests. Exceeding either threshold will result in an automatic 10 minute ban. Repeating the process three times in 24 hours results in an automatic 7 day ban for HTTP ports, 80 (HTTP) and 443 (HTTPS).

Blocked clients are returned a 403 status code (Forbidden).

Pages in total

Pages in total (PIT) log all URL requests from an IP address in a window discussed below. If an IP address exceeds that number of requests within the window, it will be blocked automatically. If a page is image heavy as verified by webpagetest.org, consider consolidating images into sprites or inlining small assets to bypass accessory HTTP requests.

Same page

Same page requests are more stringent and affect requests to the same URI. This is designed to filter out brute-force attacks. If you poll a page repeatedly, such as autocomplete with a keydown event, add a collection threshold via setTimeout that will only poll after the typist has given a momentary repose to collect thought. For instance, a simple jQuery implementation:

$("#input").on('keydown', function() {
    var timer;
    timer = setTimeout(function() {
        cancelTimeout(timer);
        // cancel other async events
        // do autocomplete AJAX callback
    }, 250 /** 250 milliseconds */);
});

This assumes that the person will type at least 4 characters per second. Words per minute is standardized to 5 characters, this it works out to be 48 WPM. You can evaluate for yourself what 48 WPM is. To avoid triggering the same-page block, without a delay (via setTimeout), one would need to type of 96 WPM with an autocomplete AJAX callback. Feasible, but unlikely.

Blocking criteria

The following thresholds are in place to filter bot from human.

Same page: 4 pages in 1 second
Pages in total: 150 pages in 3 seconds

Three blocks in 24 hours results in a seven day ban. Once a ban is in place, the only way to proceed forward is to open a ticket to remove the ban.

Leave a Reply

Let’s Encrypt behind a reverse proxy

By default, apnscp will perform an IP check to ensure a hostname maps back to the configured IP address before issuing a certificate. This is true for both initial requests and automatic renewals. Automatic renewals occur 10 days before expiration. Both the panel and API allow you to circumvent this requirement.

This does not bypass DNS propagation or domains that are unreachable via DNS. This only affects hostnames that are behind a reverse proxy such as CloudFlare or SiteLock. A challenge must still be accessible from the domain, which points to a random location on the server. This is consistent with Let’s Encrypt’s ACME server that performs the mandatory check before issuing a certificate for each hostname.

Issuance – Panel

Only certificate issues may be bypassed within apnscp. To bypass a DNS check on certificate issuance, disable the IP check option.

Bypassing DNS check for Let’s Encrypt within apnscp

Renewal – Beacon/API

The API must be used to renew Let’s Encrypt certificates if DNS bypass checks are necessary. This may change in the future. Beacon provides a frontend to the API, and for the sake of simplicity, will be used in this discussion. After configuring Beacon, access letsencrypt_renew and pass false to the optional verifyip parameter. This will disable IP verification checks that cascade into letsencrypt_request.

beacon eval letsencrypt_renew 0

Because the panel will automatically renew SSL certificates beginning 10 days before expiration, this should be done every 60-80 days. If it fails, no email will be generated, so pay heed to the return value.

To simplify operation, add a scheduled task to run monthly or bimonthly within apnscp via Dev > Task Scheduler.

Leave a Reply

Running Discourse

Discourse is a popular forum software written in Ruby. Because Discourse relies on Docker, which is incompatible with the platform, installation must be carried out manually. A Pro package is recommended to run Discourse as each worker is approximately 200 MB.

Getting Started

Installation is done within the Terminal.

  1. Checkout the Discourse repository from GitHub into /var/www
    cd /var/www
    git clone https://github.com/discourse/discourse.git
    cd discourse
  2. Verify the Ruby interpreter is at least 2.3.1. Switch interpreters if not.
    rvm list
    # Look that at least ruby-2.3.1 is the current interpreter
    rvm use 2.4.1
    • Optionally designate the Ruby version as your default Ruby interpreter for the directory:
      echo 2.4.1 > .ruby-version
  3. Install Bundle and application dependencies:
    gem install bundle
    bundle install
    • Note: depending upon platform, the build process will fail on the pg gem. Set the pg build configuration, then rerun bundle install to continue installation (v7 uses PostgreSQL 9.6, v6.5 9.4, and v6 9.1):
      bundle config build.pg --with-pg-config=/usr/pgsql-9.6/bin/pg_config
  4. Setup Redis
  5. Create a PostgreSQL database (Databases > PostgreSQL Manager). Be sure to bump the user designated to connect to the database from the system default 5 to 15 concurrent connections. Discourse pools its connections and requires a higher allowance.
  6. Create a new user to relay email for Discourse via User > Add User. Ensure that the user has email privileges (incoming, outgoing) enabled.
  7. Copy config/discourse_defaults.conf to config/discourse.conf, this will provide application-specific overrides
    cp config/discourse_defaults.conf config/discourse.conf
  8. Change the following config/discourse.conf  values to match what is on your account: db_pool (set to 5), developer_emails, db_name, db_host (set to 127.0.0.1), db_username, db_password, hostname, smtp_address (set to localhost), smtp_user_name, smtp_password, smtp_openssl_verify_mode (set to none), redis_port
    • developer_emails is a comma-delimited list of users that register who are conferred admin rights
  9. Populate the database:
    RAILS_ENV=production bundle exec rake db:migrate
    • Migration will fail requiring the hstore, pg_trgm extensions. Open a ticket in the panel to request hstore and pg_trgm to be added to your PostgreSQL database (also include the database name). Alternatively, use Beacon to call sql_add_pgsql_extension. Both “pg_trgm” and “hstore” are supported.
  10. Precompile assets:
    RAILS_ENV=production bundle exec rake assets:precompile
  11. Create an upload storage directory under public/
    mkdir public/uploads
  12. Dispatch Sidekiq, which handles tasks for Discourse, including sending email
    RAILS_ENV=production rvm 2.4.1 do sidekiq -L /tmp/sidekiq.log -P /tmp/sidekiq.pid -q critical -q low -q default -d
  13. Populate initial posts
    RAILS_ENV=production bundle exec rake posts:rebake
  14. Setup Passenger to serve the application:
    gem install --no-rdoc --no-ri passenger
    passenger-config --ruby-command
    • Add RailsEnv production to your .htaccess control under public/
    • Add PassengerRuby directive to your .htaccess control under public/, e.g.
      PassengerRuby /home/apnscp/.rvm/gems/ruby-2.4.0/wrappers/ruby
  15. Lastly, connect discourse/public to a subdomain or addon domain.

    Connecting Discourse to a subdomain named discourse.mydomain.com

  16. Visit your new Discourse forum. Register using your email address specified in developer_emails. A confirmation email will be sent if all is configured correct (smtp* settings) and Sidekiq is running. Click the link and follow the setup instructions!Note: adding a CDN in the following section is highly recommended

Adding CDN

Without a CDN, Discourse will serve all content through its application, which creates significant overhead. Placing a CDN in front of the static assets will allow a third-party to cache and send static content thus speeding up Discourse significantly.

Any CDN will work. Amazon CloudFront offers 50 GB free and will be used for this example.

Given Discourse runs on discourse.mydomain.com and the CDN will be called cdn.mydomain.com:

  • Add a CORS header to public/.htaccess:
    Header set Access-Control-Allow-Origin "http://discourse.mydomain.com"
  • In CloudFront, click Create Distribution
    • Select Web
    • Under Origin Domain Name, enter discourse.mydomain.com
    • Under Origin ID, enter cdn.mydomain.com
    • Under Alternative Domain Names (CNAMEs), specify cdn.mydomain.com
    • Under Forward Headers, select Whitelist
      • Whitelist Headers is now accessible. Under the header list, select Origin. Click Add >>.
  • Visit DNS > DNS Manager. Create a new CNAME record for cdn.mydomain.com.
    • Select CNAME as RR.
    • Enter the CloudFront Domain Name as your parameter.
    • Click Add

      CloudFront domain field

  • Edit config/discourse.conf. Change cdn_url
  • Restart Discourse
    touch tmp/restart.txt

Leave a Reply

File management with multiple users

Access control lists (ACLs) may be used in multi-user environments to allow granular joint access to file management without allowing access by all users on the account. ACLs can be established either by the owner of the file or account admin using Beacon.

ACLs come in two forms, an active entry and default. Active are actively applied to the file or directory whereas default ACL entries are applied on directories to files created in the future within that directory.

Using setfacl

ACLs may be set from the terminal using setfacl on all v5+ platforms. setfacl may only be applied on files owned by the current user. For files owned by another user, use file_set_acls in Beacon (below) or take ownership of the files first using file_chown in Beacon or chown in sudo.

Syntax to set an ACL entry is setfacl -m [d:]USERNAME:PERMISSIONS FILE where:

  • d: is an optional specifier to apply the ACLs as default ACLs rather than active ACLs
  • USERNAME is the user on the account to apply these ACLs to
  • PERMISSIONS is an octal bitmask between 0 and 7 or a collection of r,w,x representing read/write/execute permissions respectively
  • The -m … command may be repeated an infinite number of times to apply new rules to other users
  • -R may be specified to apply the rules recursively

Simple usage

$ setfacl -m user:tom:7 newdata.zip
$ getfacl newdata.zip
# file: newdata.zip
# owner: myadmin
# group: myadmin
user::rw-
user:tom:rwx
group::r--
mask::rwx
other::r--

More examples

  • Granting an additional user read access
    setfacl -m u:lisa:r file
  • Revoking write access from all groups and all named users (using the effective rights mask)
    setfacl -m m::rx file
  • Removing a named group entry from a file’s ACL
    setfacl -x g:staff file
  • Copying the ACL of one file to another
    getfacl file1 | setfacl --set-file=- file2
  • Copying the access ACL into the Default ACL
    getfacl --access dir | setfacl -d -M- dir

Further reading

Check out the man page on both setfacl and getfacl

Using Beacon

Beacon provides an alternative interface to ACLs that can run from using file_set_acls and file_get_acls. ACLs set via Beacon override traditional discretionary access checks when applied as the primary account holder; this means that as the primary user, you can alter any ACL on any file whereas using setfacl from the terminal requires that the file you are adjusting be owned by you.

$ beacon eval file_set_acls /var/www/html redline 7
1
$ getfacl /var/www/html
getfacl: Removing leading '/' from absolute path names
# file: var/www/html
# owner: myadmin
# group: myadmin
user::rwx
user:redline:rwx
group::r-x
mask::rwx
other::r-x

To set default ACLs, supply a third parameter: default:1 and to apply recursively, recursive:1

$ beacon eval file_set_acls /var/www/html/test redline 7 [default:1,recursive:1]
1
$ getfacl /var/www/html/test/foo
getfacl: Removing leading '/' from absolute path names
# file: var/www/html/test/foo
# owner: myadmin
# group: myadmin
# flags: -s-
user::rwx
user:redline:rwx #effective:r-x
group::rwx #effective:r-x
mask::r-x
other::--x
default:user::rwx
default:user:redline:rwx
default:group::rwx
default:mask::rwx
default:other::--x

To clear an ACL entry for a specific user, do not supply a permission parameter:

$ beacon eval file_set_acls /var/www/html/test redline 
$ getfacl /var/www/html/test/foo
getfacl: Removing leading '/' from absolute path names
# file: var/www/html/test/foo
# owner: myadmin
# group: myadmin
# flags: -s-
user::rwx
group::rwx #effective:r-x
mask::r-x
other::--x
default:user::rwx
default:group::rwx
default:mask::rwx
default:other::--x

Lastly, to mix and match users:

$ beacon eval file_set_acls /var/www/html/test [redline:7,apache:7]
1
$ getfacl /var/www/html/test
getfacl: Removing leading '/' from absolute path names
# file: var/www/html/test
# owner: myadmin
# group: myadmin
user::rwx
user:apache:rwx
user:redline:rwx
group::r-x
mask::rwx
other::r-x

See also

Leave a Reply

How to access WordPress admin panel

Overview

WordPress administrative panel, which allows management to your WordPress site, can be accessed within apnscp via Web > Web Apps. Select the domain where WordPress is installed. Click Detect if necessary. Click on the link under ADMIN PORTAL. If you have forgotten your login or password, click on “Change Admin Password” in Web Apps.

Leave a Reply

Sharing .htaccess rules

Overview

An .htaccess file may be shared across multiple domains and subdomains by being located in a common parent directory. Locating an .htaccess under /var/www will allow any domain or subdomain located under /var/www to inherit these rules; effectively any domain or subdomain that is not managed by a secondary user within that user’s respective home directory.

For example, assume the following layout:

  • example.com -> /var/www/example/example.com
  • sub.example.com -> /var/www/example/sub
  • blog.example.com -> /var/www/example/blog

In this situation, to have a common .htaccess shared among these 3 hostnames, locate the .htaccess file in /var/www/example.

Leave a Reply

Scripting with Beacon

Beacon is a scripting companion to apnscp that provides a simple interface to interacting with more than 2,000 commands exposed in apnscp. If apnscp can do it so can you, minus a pretty interface of course! Beacon may also be downloaded from our github repo.

Getting Started

Beacon requires an API key for authentication. Visit Dev > API Keys within apnscp to create your API key. On first use, specify --key to set the key:

beacon exec --key=AAAA-BBBB-CCCC-DDDD common_get_web_server_name

If everything went as expected, your domain will be printed out.

Once the key is set, you can skip –key=… unless you change the key again, in which case specify –key=… and –set to overwrite the key. If you’re running Beacon on your desktop to interact with apnscp, use –key=… in conjunction with –endpoint=https://launchpad.url.in.the.panel:2083/soap. Your endpoint URI is provided in Dev > API Keys.

Command Format

Each Beacon command consists of the imperative, exec, followed by the module name + method name, both in lowercase and delimited by an underscore. For example, to check system load average, which is exposed as “get_load” in the “Common” module, it is formatted as common_get_load:

beacon exec common_get_load

Which will return an array of three elements, 1, 5 and 15-minute load averages.

Parameters

Parameters, if necessary, follow command invocation. Primitives are simply passed as-is observing shell special characters and variable interpolation rules. When working with booleans, use “0” instead of “false”. Omitted parameters must be explicitly specified with an empty string (“”). Arrays may either be formatted with or without its numeric indices. Hash keys precede the value. Arrays and hashes are enclosed with “[]”, infinitely nested, and hash keys are postfixed with a colon.

Primitive examples

beacon exec sql_create_mysql_database test
beacon exec common_get_service_value siteinfo admin_user
beacon exec file_chown /var/www/myfile.txt myadmin 1

Array/hash examples

beacon exec file_set_acls /var/www/ myotheruser rwx [recursive:1]
beacon exec sql_create_mysql_database foobar
beacon exec wordpress_update_themes mydomain.com "" [avada,twentyseventeen]

Interpreting Responses

By default, Beacon presents itself in human-readable format with print_r. Use --format=json to output the result as JSON and as bash-friendly arrays, --format=bash:

$ beacon exec --format=json common_get_load
{"1":"0.52","5":"0.82","15":"0.94"}
$ beacon exec --format=bash common_get_load

(['1']='0.47' ['5']='0.81' ['15']='0.94')

No trailing EOL marker is included in the actual output for ease of parsing.

Chaining

Chaining is the real magic in Beacon. By mixing shell data types and Beacon, it’s easy to store output and work with it.

Checking if all WordPress domains are current

Iterate over all domains. Check each domain if WordPress is current, update if not.

# () also works in lieu of declare -a... declare -a DOMAINS=$(beacon exec  --format=bash aliases_list_aliases) for domain in ${DOMAINS[@]} ; do     STATUS=$(beacon eval wordpress_is_current $domain)     if [$STATUS == 0]; then continue ; fi     beacon wordpress_update $domain  done

Create an addon domain, then install WordPress or Drupal onto it

Creates a new addon domain, then installs WordPress, Drupal, or any web app. “${n,,}” takes the parameter and converts to lowercase in bash v4+ using case modifiers.

function newdomain {
    domain=${1,,}
    app=${2,,}
    path=/var/www/${domain}
    beacon exec aliases_add_shared_domain $domain $path
    beacon exec aliases_synchronize_changes
    beacon exec ${app}_install $domain 
}

Introspection

Sometimes the publicized name isn’t too clear. Use show for code introspection. Introspection fetches code from our GitHub repository and uses reflection to get an accurate representation.

beacon show common_get_load

/**
 * array get_load (void)
 *
 * @privilege PRIVILEGE_ALL
 * @return array returns an assoc array of the 1, 5, and 15 minute
 * load averages; indicies of 1,5,15
 */
public function get_load()
{
        $fp = fopen('/proc/loadavg', 'r');
        $loadData = fgets($fp);
        fclose($fp);
        $loadData = array_slice(explode(" ", $loadData), 0, 3);
        return array_combine(array(1, 5, 15), $loadData);
}

Leave a Reply

Forwarding a web site elsewhere

Overview

A forwarded website can be accomplished by first creating a subdomain or addon domain in the control panel, then using an .htaccess in document root to redirect all traffic to the new web site using mod_rewrite.

Important terminology

  • Forwarded domain: domain that will redirect to the target domain
  • Target domain: domain that is the final destination of the forwarded domain
  • Path capture: take the path in the url, e.g. /foo/bar/baz in http://example.com/foo/bar/baz, and transfer that path to the target domain. This is accomplished with a regex capture

Direct Forward, No Path Capture

Add the following lines to your .htaccess file in the document root of the forwarded domain (or subdomain) that you would like to redirect. Once a user accesses the website, the browser location will change to reflect the target domain.

RewriteEngine On
RewriteRule ^ http://DESTINATIONSITE/OPTIONALURL [R=301,L]

Direct Forward, Path Capture

RewriteEngine On
RewriteRule ^(.*)$ http://DESTINATIONSITE/$1 [R=301, L]

Proxied Forward, Path Capture

A proxied forward will attempt to keep the original domain in the Location bar of the browser and impersonate the target site as if it were hosted under the forwarded domain. There are several limitations to this approach that are only resolvable by manipulating the stream as it comes over the wire, which is also beyond the scope of this article but accomplished with reverse proxy middleware as is used in apnscp to redirect cp.apnscp.com to each platform’s control panel.

RewriteEngine On
RewriteRule ^(.*)$ http://DESTINATIONSITE/$1 [P,L,QSA]

A proxied forward will leak the target domain’s location if the target domain uses absolute URLs or sends a “Location:” header as if forwarding in one of the two above examples.

Redirect Codes

Each redirect in the aforementioned examples uses “301”. There are 4 types of redirects to familiarize yourself with, each code is pragmatic in that it instructs the browser or search engine how to handle the response:

  • 301: Permanent redirect. Clients making subsequent requests for this resource should use the new URL. Clients should not follow the redirect automatically for POST/PUT/DELETE requests.
  • 302: Redirect for undefined reason. Clients making subsequent requests for this resource should not use the new URL. Clients should not follow the redirect automatically for POST/PUT/DELETE requests.
  • 303: Redirect for undefined reason. Typically, ‘Operation has completed, continue elsewhere.’ Clients making subsequent requests for this resource should not use the new URL. Clients should follow the redirect for POST/PUT/DELETE requests.
  • 307: Temporary redirect. Resource may return to this location at a later point. Clients making subsequent requests for this resource should use the old URL. Clients should not follow the redirect automatically for POST/PUT/DELETE requests.

Credit: Bob Aman, StackOverflow

See Also

Leave a Reply

Connection to mail over SSL fails

Overview

IMAP, POP3, and SSL that connect over SSL either via STARTTLS on port 143/110/587 or 993/995/465 respectively fail with a certificate warning without any symptoms prior to October 25, 2016. Symptoms include the following dialog from Thunderbird:

SSL certificate rejection on initial connection

SSL certificate rejection on initial connection

SSL certificate mismatch inspection after clicking "View" in the "Add Security Exception" dialog

SSL certificate mismatch inspection after clicking “View” in the “Add Security Exception” dialog

Cause

With the proliferation of free SSL certificates via Let’s Encrypt, vendors have begun to tighten requirements on SSL certificate validation to thwart hackers. Thunderbird and Mail (iOS) now require that the mail server name match a name in the Subject Alternative Name extension. Without such match the aforementioned warning is generated.

Solution

Change your mail server name, both incoming and outgoing, to match the server name on which you are hosted. In the initial example, “mail.futz.net” would be changed to “luna.apnscp.com“.

Thunderbird

See KB: Manual Account Configuration

Outlook

See KB: Change email account-settings

Additional Notes

This has been corrected in account provisioning as of October 26, 2016.

Leave a Reply