26 Nov 2019

All the Gritty Details of our new Server Platform

Disclaimer

This post has been a work in progress since mid January of 2019.  Sometimes these things take some time to hammer out, and sometimes, you’re still constantly improving and refining a thing, and keep telling yourself “I’ll just wait until the post is 100% accurate and complete, and then I’ll publish it”.  And eventually, sometimes you need to just acknowledge you’re trying to describe a moving target, and really just let things go and ship the post.

Overview

2019 has been a big year for us, and it all started with the rollout of a new ‘baseline server platform’ for the technology that powers our Shared Linux Hosting Servers.  Now, to be fair quite a bit of it isn’t exactly new technology, but rather, new to us.  One of the simple facts of running a stable hosting platform is, well, that key word.. “stable”.  We don’t get to play with the latest and greatest technological advances (at-least not on our production platforms), because above everything else, we need our platform to be stable, and frequently in tech, “new shiny” tends to mean the opposite of “stable”.  Often stability comes with usage, refinement, and generally, age of the technology in question.

But we’re geeks at heart, and we love to play with the new shiny.   New beta release of RHEL?  We’ll be there poking at the first chance we get, but when it comes to production systems, we tend to be a little gun shy of brand new releases. In fact, we generally won’t run a new OS anywhere in production until after the .1 , or sometimes .2 release comes out.  It’s just something we’ve learned over time is the best thing for everyone’s sanity.  But still, we keep tabs on all the latest shiny, and when it reaches the point that we’re comfortable rolling it out into production, we always enjoy doing so.

A Year of Enhancements

Man sitting at a retro looking computer command center

What our moms think we do

Earlier this year, as part of our big server migration, we established a new ‘baseline’ for how all of our shared linux web hosting servers are setup and run.  Included in that new baseline are a number of new (for us, in production) technologies, and we want to take a moment to acknowledge some of them and what they’re going to mean for everyone.

CloudLinux

We moved from CentOS over to CloudLinux as our basic underlying operating system.  I would best describe CloudLinux to someone as “CentOS, but with added features to improve the security, stability, and performance for web hosting”.

Some of the new CloudLinux features are strictly back-end, things to make our lives easier, like the KernelCare feature, which allows us to patch the running Linux Kernel for security vulnerabilities, in real time, without having to reboot the server.

Other CloudLinux features however, have a direct and easily recognizable benefit to our clients:

  • LVE Management and CageFS – Cloudlinux provides what they refer to as a “Lightweight Virtualization Environment”.  Basically, it allows us to track/limit the amount of CPU, RAM, and other server resources down to the user account level.  This means we can not only prevent a single user from monopolizing an entire server, or even crashing it, but it also allows us to expand the types of workloads we can safely allow clients to run under the ‘LVE wrapper’ of their account.  At the same time, CageFS allows each user to have their own ‘containerized file system’, similar to a chroot jail of old, but, securely.   The introduction of LVE and CageFS and the performance/safety they brought to the table cleared the way for a number of new features we were able to roll out in 2019:
  • PHP Selector and HardenedPHP – Cloudlinux not only provides a way, via the PHP Selector, for us to offer multiple versions of PHP for client use, but they also continue to back-port critical security fixes into some older versions of PHP that are no longer officially updated by the PHP community.  This has allowed us to continue to offer a secure version of PHP 5.6 past its official end of life date, for those clients who are not yet ready to jump onto the PHP7 bandwagon.
  • Python Selector – The Python selector, in combination with LVE, allows us to offer up (as of today) six different versions of Python for client use.  This allowed us to offer Python Hosting without having to choose a side in the great “Python2 versus Python3” debate that seems to consume everyone.  We offer both, and at this point would suggest Python3, but both are there for your use.

Apache Worker MPM, FPM-PHP, and LSAPI

So, the short version is that we got some really nice performance and security improvements in what amounts to the most basic and central service to our hosting platform, the act of actually serving up web pages.  This was such a big improvement that we did an entire blog post back in June just to talk about how awesome it is.  If you want the full story, by all means, go enjoy that one. 😉

The Path Forward From Here

Right now we have the basic tech stack of our hosting platform about where we want it to be in terms of features and capabilities, so I suspect things will be  a little more quiet in that area for a little while at least, but we’re always off tinkering with new ideas and new technology.  So as those technologies firm up, get baked a little firmer and start to turn into things we can build a business offering on top of, expect to see additional new and exciting features/options coming from us in the months ahead.

01 Nov 2019

A Farewell to Sitebuilder

RVSiteBuilder Screenshot

RVSiteBuilder Screenshot

Just under eleven years ago we rolled out RVSitebuilder as a new feature to our shared linux web hosting accounts.  At the time we had, of course, high hopes that clients would find it of value.

Over the years, folks have tinkered with it, we’ve seen at best estimate around 2% of all clients ‘poke at it’ to see what it could do for them, or perhaps just to get design inspiration?  But we’ve never actually seen much traction in the way of clients actually utilizing the functionally long term, but as it was a fairly cheap ‘addon’ that we had bundled in with our cPanel licenses via the cPanel licensing provider we use, we never really gave much thought to it beyond over the years from a ‘cost/value’ perspective.

But as regular readers may be aware, there was something of an upheaval with regards to cPanel licensing announced over the summer.

Not quite as widely known was that in the fallout of that pricing model change, one of the larger cPanel licensing vendors, well, they sold their entire operation.  Then, a little while later, the new owners basically announced that that were terminating some partnerships and a bunch of products you could previously license through them would no longer be available.

After a little bit of a scramble, we realized that, on top of the recent tripling of our per server licensing costs due to the cPanel increase, we could be facing a second round of increases for all of the ‘non-cPanel’ items we use alongside of cPanel on each server if the product was one being discontinued from our provider.

Short version?  The per-server cost of RvSiteBuilder would be tripling if we choose to keep it around.  After once again evaluating just how much usage it was getting, this week we’ve disabled it across all our servers, and are retiring the functionality.

We are currently evaluating some other, more modern site builder options for our client’s use, and hope to once again have a easy to use site builder option available in the future.

04 Oct 2019

Upcoming Maintenance Windows

We’ve recently been notified by one of our data center providers that they need to schedule some emergency maintenance windows in order to apply some critical updates to the underlying physical hardware, and that these updates will require taking some of our servers offline while they perform the work.

While we know that these scheduled events are never ideal, we have <knock on wood> been very lucky in terms of infrastructure outages since moving from our own managed hardware into provider-managed cloud servers.

We’ve been collating the scheduled windows and are currently:

  • notifying all impacted clients via email of their scheduled outage window.
  • adding the impacting events into the Network Status / Scheduled Events section of our client portal.
    • This is particularly cool to us, as it allows us to input all future scheduled events, and clients will only see/be bothered by the events that actually impact them personally.  Yes, we’ve had the ability to do the same via email for some time, but this is just another cool feature of the ‘new’ client portal that we’re really getting to utilize for the first time.

We will continue to monitor the situation and should any further windows be scheduled by the provider, we will of course notify additional impacted users and add those events to the client portal as well.

 

 

12 Sep 2019

Node.js support is now live!

man holding a small sign that reads 'node js'This evening we’re taking the covers off our latest feature addition to our Shared Linux Web Hosting plans, everyone please welcome Node.js to our feature list!

The Node.js setups is very similar to the Python support we unveiled earlier his year, in that you can deploy a new Node application into your account by looking for the “Setup Node.js App” icon under the “Software” section in cPanel. You’ll be prompted to select the Node version you wish to use, where the application should live folder wise, and the URL you want to associate with the application, cpanel takes care of all the rest.

At this time the Node.js selector will default to v10, but you can also choose to run your applications under v12 if so desired.

17 Aug 2019

More details on the cPanel price increase

So, the great cPanel Price Increase of 2019people counting coins on a table is scheduled to go into effect in just over two weeks, and, not so shockingly, details are still a bit slow in coming.  Officially the word we’ve seen from our license distributor is “The pricing you saw previously is valid, that’s what we’ll be charging as of September 1.  As for any sort of volume discount or any details beyond what was in the original announcement, we’re still waiting for that information ourselves.

Short version for us is, that as of right now, cPanel licensing goes from approximately 6% of our cost per server, to about 18% of each server.   This does not include CloudLinux, Fantastico, or any other software we license for our server fleet, just cPanel itself.   cPanel by itself will, after September 1, cost us more per server each month by itself than what we pay for all the software we license for a given server today.

The good news for our clients is, we’ll be absorbing that cost directly.  We have no web hosting plan pricing changes scheduled or planned as a result of the cPanel cost increase.

Long term, we’re honestly still a bit worried about what this pricing model change is going to mean for cPanel in relation to the industry as a whole.  cPanel has long maintained a market dominate position simply by virtue of being the “best of the cost effective control panels”…  Starting September 1, they’re kind of moving themselves out of the “cost effective” category for a number of business models, and based on the rumblings we’re hearing inside the industry, we suspect you may see a spike in popularity and/or new Control Panel options entering the market.

For our part, we’re going to keep an eye on the situation as it evolves and see how it all shakes out.  But in the meantime we’re maintaining course for our clients.

18 Jul 2019

PHP 7.3 now available

Screenshot of PHP code in a text editor

PHP 7.3 has been rolled out across all of our Linux servers, and now available for general use.

To select PHP 7.3, you will need to change your selection using the “MultiPHP Manager” which can be found inside your cPanel interface under the “Software” section.

The PHP 7.3 Migration Guide may prove useful and provide a quick reference of what has changed within the language.

29 Jun 2019

cPanel Increases Pricing, Potential Chaos Ensues

Yesterday the folks @ cPanel lobbed something of a grenade into the middle of the web hosting industry with a blog post entitled Announcing Account Based Pricing.

Long story short, they are increasing the cost of cPanel for pretty much every situation under the sun effective September 1st. The old pricing model was a flat monthly fee per licensed hosting server, while the new pricing is directly tied to the number of web hosting accounts on the server.

Now, this announcement has sparked a number of reactions, including anger, confusion, and fear from around our industry. We’ve seen reports of providers who believe they will see massive increases in their cPanel costs going forward, with some expressing concern of just how they are going to make the numbers work. These conversations have lead to discussions of providers moving to other platforms, cutting ancillary features out of their offerings to lower costs, or simply closing up shop as they will no longer be able to financially stay above water given the new pricing model.

Some of those reactions and discussions have filtered out from the industry watering holes and into the webmaster forums and chat rooms, to the point we have had more than a couple clients reach out in the last 24 hours asking for information on the topic from our perspective.

With that in mind, even though we don’t yet have 100% of the picture (the pricing being bandied around so far is ‘retail’, we’re not yet sure exactly where our final pricing will land), I wanted to take a moment to discuss both the big picture, as well as what it means for us here at Pure Energy.

In the big picture, things could get dicey for some of the lower cost hosting providers over the next couple months. The lower end market is run on razor thin margins, and, well, I’ll be blunt, it’s hard to stay afloat selling $5/year hosting if your control panel alone is costing you $4/year per account.

For our own environment and business model, we’ll see around a 3x increase in our monthly cPanel license costs, at the retail pricing we’ve seen so far. While not ideal for us, especially coming so closely on the heels of the cost increase we absorbed just a few months ago with the addition of CloudLinux to our servers, at this time I believe we can make the numbers work for our existing Shared Linux Web Hosting plans without a price increase, and without any other substantial changes to our service, plans, or features.

We won’t have 100% of the picture until we get firm pricing from our provider, but to be honest, the bigger concern for us is the uncertainty this injects into the idea of cPanel as a stable platform for our future:

  • Our biggest, as of yet unannounced, project for 2019 was a ‘scratch our own itch’ service that we were hoping to integrate into cPanel pretty tightly and offer to our users as a value added feature of hosting with us, as well as to market the service to other hosting providers as a way to scratch their itch as well. This is on hold (the integration, not the service!) while we see how the cPanel landscape shakes out.
  • A smaller project we had been looking at involved clients who find themselves needing much larger resource allocations than any of our current plans offer. The idea here was larger servers, larger plans, fewer clients per server. We have to re-run the numbers there to see if cPanel still makes sense once we have the finalized pricing.
  • cPanel’s immense popularity has long made it the de-facto ‘go to’ for any provider/service/platform that had a product of interest to the hosting industry. The fallout from this price increase, if it serves to fracture the landscape of providers, could very well jeopardize the dominance that cPanel currently enjoys in the market. If that happens, what happens to the 3rd party interest/support for cPanel?

Some of these are obviously questions that wont be answered for some time, for right now however, things, for us, are status quo. We’ll keep doing the best we can for our clients, keep our eyes on the industry landscape, and keep moving forward.

16 Jun 2019

The Evolution of PHP under Apache

Over the sixteen years we’ve been doing this web hosting thing, we’ve seen a multitude of changes to the way webservers run, and more notably, in how they run scripts under, for instance, languages such as PHP.

In the beginning: PHP as CGI

In the beginning running a PHP script was a fairly simple affair.  The Apache webserver of the day used a preforked child based process for handling requests, where the main web server had a pool of child processes, and as each web request came into the server, it would be allocated to a worker, the worker would process that one request, and if a PHP script was involved, a separate CGI process would be spawned out to run PHP and process the scripting part of the page.   There would be a whole slew of settings in your apache configuration that would boil down to “PHP files located in /home/bob/public_html/ get run under the ‘bob’ user account”.

This worked, but it was anything but fast.  As time went on and PHP became more popular, and the sites that it powered became more complex, performance became a big issue, in order to keep things efficient and cost effective, a better way had to be found.

Enter mod_php

Along came mod_php, which sort of up-ended the entire apple cart compared to how we all thought of PHP.   mod_php was an actual module that you linked into Apache itself.  Now your PHP scripts were basically interpreted by the webserver.  No longer did Apache have to ‘spawn out’ another process to render your PHP content.  This was fast.  Then OpCode caching became a thing, with add-ons such as the APC module, where now you could allocate some RAM to be used to persistently cache the PHP code (and possibly other data) in between requests.  Things got extremely fast, and there was much rejoicing on the parts of web hosts everywhere as CPU usage decreased on servers everywhere.

Downside, this meant all PHP scripts needed to run as the same user that Apache ran as.  This meant that everyone’s scripts ran as “nobody”, or possibly “apache”.  Having all the PHP scripts on a server that hosts multiple clients and websites all run under the same security user account was (although I don’t believe anyone initially saw it coming at the time) a huge problem.  The industry started to see a rise in the number of defaced and hijacked sites, and it quickly became evident that the “everything runs as nobody” was to blame.

Malicious actors quickly built PHP scripts that when run, would scour a server, for instance by looking for any *.php files in /home/*/public_html/ on the server, and every time it found a php file it could write to, it would then inject whatever malicious code it desired to spread.  Since all the php scripts on a server ran as the same user, that user account would, by default, need access to all the scripts on the server.   It only took finding one single exploitable script on a website in which they could launch their script to allow the malicious actor to infect (or delete) all the php scripts on the server.  This of course, was not a good solution, as now every script and client account on your server was only as secure as the most out of date, un-patched script on the server!  Something else would need to come along…

(Side Rant:  If, in 2019, you find a web host that is still suggesting that you chown your scripts to “nobody:nobody” or “apache:apache”, be very afraid.  The reasons for this fear should be evident from the last paragraph, but I feel the need to re-iterate the point, because doing so is, well, just a huge mistake security wise!)

SuPHP to the rescue

Then along came SuPHP, which combined most (but not all) of the performance benefits of mod_apache with the flexibility of scripts once again being able to run as individual user accounts.   It was something of a balanced trade off between “fast” and “secure”.

Apache Gains Workers, But They’re Useless To Us

Along with other great advances to the Apache Web Server, eventually it gained the ability to have ‘Workers’ instead of just ‘Preforked Children Processes’.  While child processes worked fine for years, the overhead of ‘every connection has a dedicated child process that feeds it’ involves a fair amount of overhead.

So the Apache team implemented the ‘Worker’ MPM module.  The server still forks a group of child processes in advance, but each child process is capable of serving multiple http connections via Threads.

This brought a sizeable boost in performance on hosting servers (notably a decrease in memory needed to service a given number of http connections), but it had one big drawback… it didn’t work with PHP, so it was no good to us for quite some time.

mod_lsapi:  The best of all worlds?

With the recent migration of our servers over to the CloudLinux platform, we’ve gained the ability to leverage the lsapi Apache/PHP interface to achieve a modern ‘best of all worlds’ approach:

  • Apache runs with the mpm_worker module, allowing one child process to handle multiple connections via threads.
  • lsapi interfaces those workers to php via php-fpm as needed when an php file needs to be processed.
  • php-fpm gives us flexibility, security, and performance all in one.

Wait, what’s PHP-FPM?

PHP-FPM is sort of a ‘php server’ if you will.  It is responsible for maintaining a pool of PHP processes that can be handled your PHP code to process.

The flexibility comes in that we can control the version of PHP on a per site level, which is how we can provide the ability for users to control which version of PHP their code runs against.

On the security front, each user has their own self contained php server process, providing a ‘pool’ of PHP processes at their disposal, each running as that user, so no worries about ‘nobody’ permissions or users being able to access each other scripts.

For performance, well, we ‘persist’ the pools per user for a set amount of ‘idle time’ after the last PHP script was processed.  This means that while on the first PHP request for a given client site, we need to (quickly) spin up a PHP process, that process will live-on after the request is finished (up to our idle timeout), if a second request comes in before that time-out ends, it is serviced by the same process, saving start up time, and more importantly, allowing us to once again benefit from the performance boost of PHP OpCode Caching.

The ability to use a PHP pool, when combined with the per-user resource scheduling available to us in CloudLinux, really opens things up possibility wise, allowing us to fairly distribute resources while maintaining performances across all users on a given server.

For instance, here’s the CPU and RAM utilization over a given day for a typical user on one of our shared linux webhosting servers:

cpu and memory graph showing normal traffic activity

Now, this users site got a pretty normal consistent level of traffic throughout the day, it of course had some spikes here and there, but overall, pretty even-keeled distribution of traffic.   You’ll note that the memory usage is pretty even throughout the day, even though CPU jumped a bit more here and there..  This is because even though the clients site may have needed more processing time, the standard PHP process pool for them was already running, and simply kept going handling the visitor load.

Now, lets see what happens when things get… busy.  This customer had a far more interesting day:

cpu and memoryg graph showing peaks caused by a slight DDoS attack

Now, this graph is a bit of an anomaly, not enough to cause a panic, but, well, they were dealing with a slight DDoS attack, in for the form of someone attempting to brute force user logins to their forum.   The traffic was pretty steady throughout the day, but with large spikes in traffic at random times along the way.

While CPU utilization got pretty intense at times, RAM utilization stayed pretty consistent (most of the time)..   there were a couple times throughout the day when the system determined it was unable to provide all the cpu resources demanded by the attacker, so things did get throttled a bit here and there with the ebb and flow of traffic coming in from the attacker, but the ability for the server to maintain a single, persistent pool of PHP processes to handle the requests, complete with OpCode caching and all the other benefits of reusing the same process for multiple requests, allowed the overall impact to memory to stay pretty consistent throughout the day.

The important thing in this scenario is, that with the persistent php process and OpCode caching being available between requests, the overall load on the server generated by this attack was minimized.  The system did it’s best to maintain service to the client’s website throughout the ‘bursty’ attack periods, and no other client sites hosted on the same server saw any impact at all.

This is a huge change from the days of ‘PHP as CGI’, and we’re confident that Apache Workers + mod_lsapi + php-fpm gives us just the right combination of security, flexibility, and performance going forward to best serve our clients.

 

17 May 2019

Git: Now available across all our hosting plans

logo of the git software projectWe’re happy to announce the immediate availability of Git repository hosting as a standard feature for all of our shared hosting linux plans.

Git is a distributed version control system, allowing you a safe and effective means of tracking revisions and edits to your source code.  The predominate use-case for Git by our clients would be, of course, storing and tracking the web code that powers your site, but I’m sure some enterprising clients will find other, additional uses for it.

The Git repository management section of cPanel can be found under the “Files” section of your cPanel interface, via the icon labeled “Git Version Control”.

The deployment of git repository hosting builds on-top of the same security and resource management feature additions that enabled us to bring about the return of secure shell access previously.  Research (disk/cpu/bandwidth) usage via Git is shared and accounted for within your account’s base cPanel allocations.  (In other words, if your Git repo is 50MB in size, that 50MB of disk space will count against your accounts normal Disk Quota allotment.)

(c) 2019 Pure Energy Systems LLC - All rights reserved.

back to top