Chris Hope's LAMP Blog - The Electric ToolboxChris Hope's LAMP Blog - The Electric Toolbox

Linux Apache MySQL and PHP articles by Chris Hope

This is Chris Hope's blog for Linux, Apache, MySQL and PHP (known as LAMP) and Javascript/jQuery. I started this website several years ago with articles about web programming, Linux and Windows tips and tricks, howtos etc.

The ten most recent articles can be found below in their entirety. Navigating the sections in the right navigation (under Categories) will bring up all the other posts, and you can also use the search box at the top of the page to find what you might be looking for.

curl error [globbing] bad range specificationcurl error [globbing] bad range specification

Posted June 26th, 2015 in Linux/Unix/BSD

When testing some website redirects with curl from the command line, I got the error message "curl: (3) [globbing] bad range specification in column 120"

Escape [ and ] or use -g or --globoff

An example URL that will cause the error is this:

curl "http://www.example.com/?test[]=123"

Either escape the square brackets like this:

curl "http://www.example.com/?test\[\]=123"

or use --globoff like this:

curl --globoff "http://www.example.com/?test[]=123"

or the shorter -g like this:

curl --g "http://www.example.com/?test[]=123"

ERROR 1115 (42000) : Unknown character set: 'utf8mb4' in mysqlERROR 1115 (42000) : Unknown character set: 'utf8mb4' in mysql

Posted June 25th, 2015 in MySql

When attempting to load a database dumped with mysqldump into an older MySQL database, I got the error message ERROR 1115 (42000) : Unknown character set: 'utf8mb4' in mysql.

What's utf8mb4?

utf8mb4 is a utf8 character set, which was added to MySQL in version 5.3.3, that fully supports unicode. Read this post by Mathias Bynens for more information about the difference between to the two character sets with MySQL.

How do I solve the problem?

If you dumped a table/database that supports utf8mb4 then you need to load it back into a MySQL database that is at least version 5.3.3. If you try to do it in an older version, you're going to get the error message I gave in the opening paragraph.

A quick and dirty solution, not recommended

There is a way to workaround this, but I strongly advise not to, as you may cause data loss and/or other unexpected consequences. In my case, I was simply trying to benchmark database queries running across different servers with the same database, so it didn't really matter too much if some utf8 characters got saved incorrectly or other content modified.

Let's say you dumped the database like this:

mysqldump -u [username] -p [database] > database.sql

You can then run "sed" against it to change all the instances of utf8mb4 to utf8 like this:

sed s/utf8mb4/utf8/g database.sql > database.sed.sql

It'll run pretty fast; doing this on a 500MB file only took maybe 10 seconds on the server I was doing it on. Note that it will replace all instances of utf8mb4 with utf8. So, for example, if I'd dumped the table containing this post then all instances in the content would also have changed.

You can probably modify the sed expression to work out all the various ways the character set is set in the file and run those, avoiding modifying anything else, but you still may have issues with character encoding being broken - just a final warning :)

A better solution?

If you have a better solution, feel free to add it in the comments.

Importing YouTube into WordPress from another CMSImporting YouTube into WordPress from another CMS

Posted June 17th, 2015 in PHP

I've been involved in converting a custom CMS to WordPress and we imported the content (mostly) successfully into WordPress, but the YouTube videos wouldn't embed, even when stripping out the iframe and just leaving the YouTube URL in the content.

How it normally works

You would normally just copy and paste a YouTube URL into WordPress and it will automatically do all the clever stuff to add whatever tags are necessary to embed it into the page. Then you'd save it and, hey presto, it all works like magic.

But when importing and the YouTube video is in an iframe

If you assign HTML to post_content and save it using wp_update_post(), it completely strips out the iframe, and won't even leave the YouTube URL there. Whoops.

But when importing and the post_content contains HTML

If you strip out the iframe and just leave the YouTube URL, for example like this:

$content = preg_replace('~<iframe.*"(https?://www.youtube.com/.*)(\?rel=0)?".*</iframe>~U', '\1', $content);

but there's other HTML in post_content, then WordPress doesn't seem to want to do its magic with YouTube URLs. Whoops.

I found that if you enclose the YouTube URL with [embed] ... [/embed] then it works. I have seen suggestions to do [youtube=...], but that didn't work for me.

So, revising the above regular expression to this:

$content = preg_replace('~<iframe.*"(https?://www.youtube.com/.*)(\?rel=0)?".*</iframe>~U', '[embed]\1[/embed]', $content);

did the trick for me.

I hope it works for you too!

How to do a tick/check and a cross in ExcelHow to do a tick/check and a cross in Excel

Posted May 11th, 2015 in Applications

There are several fonts which let you easily type a tick/check or cross into a cell in Excel, or use in any other application; or you can use unicode.

Using fonts

Use the Marlett or Webdings fonts, and use "r" for a cross and "a" for a check or tick.

Using unicode

There are a few checkmarks and crosses that can be used with unicode too.

U+2714 is "heavy check mark"

U+2718 is "heavy ballot x"

This page at Wikipedia shows how to put unicode into a document: http://en.wikipedia.org/wiki/Unicode_input

Xamarin Android Player to simulate Android devicesXamarin Android Player to simulate Android devices

Posted May 7th, 2015 in Applications

Five years ago, I bought a Mac Mini so I could use the iOS Simulator (and then quickly "upgraded" to a more powerful MacBook Pro when I realised I far preferred OS X to Windows) but struggled to find a decent equivalent for Android devices. After a quick Google today I found the Xamarin Android Player, which does the trick nicely.

Xamarin Android Player

Download it from here: https://xamarin.com/android-player

There are versions for both Windows & OS X. I've only tried the OS X version. You download and install it, and then there's a list of downloadable devices. At the present time, these are:

  • Nexus 4 KitKat, Lollipop & Jelly Bean - 768x1280
  • Nexus 5 KitKat & Lollipop - 1080x1920
  • Nexus 7 KitKat, & Jelly Bean - 800x1280
  • Nexus 10 KitKat - 1600x2560 (i.e. 800x1280)
  • Nexus S KitKat - 480x800

I'm not sure how or if you can add other images.

The only annoyance I have found so far is that the virtual device will only use DNS for domain lookups, and won't look at your local hosts file (whereas the iOS Simulator will use the hosts file first, and DNS lookups next).

Set PHP configuration options with NginxSet PHP configuration options with Nginx

Posted April 26th, 2015 in Nginx Web Server and PHP

There may be times you need to set specific PHP options for a virtualhost on Nginx, for example adding an automatically prepended file to all pages. This post shows how to do this.

PHP auto_prepend_file example with Nginx

Let's say you have a setting like this to make all .php files be parsed as PHP:

server {
  ... configuration options ...
  location ~ \.php$ {
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_param	QUERY_STRING		$query_string;
    fastcgi_param	REQUEST_METHOD		$request_method;
    fastcgi_param	CONTENT_TYPE		$content_type;
    fastcgi_param	CONTENT_LENGTH		$content_length;
    ... etc etc ...
  }
}

Simply add this in with the other fastcgi_param values:

fastcgi_param PHP_VALUE auto_prepend_file=/path/to/prepend.php;

The above example will automatically prepend the file at /path/to/prepend.php to all files that have a .php extension and parse them as PHP.

You should be able to set any other PHP value that can be configured outside the php.ini file in the same way, changing auto_prepend_file to the confguration name, and /path/to/prepend.php to the configuration value.

Fix "access denied" error when parsing HTML as PHP with NginxFix "access denied" error when parsing HTML as PHP with Nginx

Posted April 25th, 2015 in Nginx Web Server and PHP

If you are attempting to get files with a .html extension parsed as PHP with Nginx + php-fpm, you might get an "access denied" error in your browser, and the error message "Access to the script '...' has been denied (see security.limit_extensions)" in your Nginx error log. This post shows how to allow html files to be parsed successfully as PHP with Nginx + php-fpm.

Nginx configuration block

Your Nginx configuration block will look something like this to parse HTM files as PHP:

server {
  ... configuration options ...
  location ~ \.html$ {
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_param	QUERY_STRING		$query_string;
    fastcgi_param	REQUEST_METHOD		$request_method;
    fastcgi_param	CONTENT_TYPE		$content_type;
    fastcgi_param	CONTENT_LENGTH		$content_length;
    ... etc etc ...
  }
}

Error message

And yet when you access e.g. somefile.html you get "Access denied" in the browser.

You check the Nginx error log and see this:

2015/04/25 07:38:28 [error] 5942#0: *108814 FastCGI sent in stderr: "Access to the script '/path/to/somefile.html' has been denied (see security.limit_extensions)" while reading response header from upstream, client: 192.168.1.54, server: www.example.com, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "www.example.com"

The fix needed is suggested in the error message: "see security.limit_extensions"

Set security.limit_extensions

On Debian 7 Wheezy, the configuration file to edit this setting is at /etc/php5/fpm/pool.d/www.conf; on other distributions it may be in a different place.

Then search for security.limit_extensions. It should look something like this in the file by default:

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
;security.limit_extensions = .php .php3 .php4 .php5

If the security.limit_extensions has been set already, then add .html to it; if it hasn't then add it in with all the extensions you need to allow, e.g.:

security.limit_extensions = .php .html

Is it safe to do this?

When enabling this myself, my first thought was "Is this safe" and "Can a regular HTML file suddenly be parsed as PHP" and then obviously "Will this cause security issues with any WordPress blogs installed on my server?"

As far as I can tell, it shouldn't cause any issues, because you still have to allow .html files to be parsed through php-fpm in the Nginx config. If you haven't done that, then they won't.

If I am wrong, please add a comment below.

How to shrink a VMWare Fusion Linux virtual diskHow to shrink a VMWare Fusion Linux virtual disk

Posted April 21st, 2015 in OSX and VMWare

VMWare Fusion on Mac OSX allows you to shrink virtual disks easily for Windows virtual machines right from the graphical interface (Virtual Machine -> Settings -> General -> Clean Up Virtual Machine) but there isn't the same option to do this with Linux, so you need to do it from the command line.

1. Clean up space

First of all you should clean up whatever space you can on the Linux virtual machine by deleting uncessary files, etc. After doing that, run this command as the root user on the Linux VM to zero out any free space - if you don't do it, then the shrinking process won't reclaim any space:

cat /dev/zero > zero; sync; sleep 1; rm -f zero

You'll see this message when it has finished:

cat: write error: No space left on device

That's OK, because the idea was to fill the disk up and then delete the file afterwards.

2. Shut down the virtual machine

You can't reclaim space with the virtual machine running, so shut it down first.

3. Run the shrinking process

Open up Terminal on your Mac, change to the directory the virtual machine files are in (for example I have mine at ~/Virtual Machines/<vm name>) and then run the following commands, replacing "Virtual Disk.vmdk" with the actual name of your virtual disk image.

Note that if the filename contains spaces then the filename parameter to the command needs quotes around it, as in my examples.

The following examples assume VMWare Fusion is installed in Applications at the OSX disk root.

The first command defragments the disk:

/Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -d "Virtual Disk.vmdk"

I got the message "VixDiskLib: Invalid configuration file parameter. Failed to read configuration file." when running this command. I don't know what this error means, but the command worked successfully and this was the other output:

Defragment: 100% done.
Defragmentation completed successfully.

The second command then shrinks the disk:

/Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -k "Virtual Disk.vmdk"

Once again, I got the message "VixDiskLib: Invalid configuration file parameter. Failed to read configuration file." but again it all worked fine, and this was the output

Shrink: 100% done.
Shrink completed successfully.

I managed to reclaim 9GB of wasted space from doing this.

What is the virtual disk filename?

When you do a directory listing, you'll see a bunch of files and directories like this:

564d0102-8ca8-949e-169e-7f39e33745b7.vmem
564d13fe-4467-4a08-e14b-10c083090f70.vmem
564d13fe-4467-4a08-e14b-10c083090f70.vmem.lck/
Debian 7 64-bit-03868d7d.vmem
Debian 7 64-bit.nvram
Debian 7 64-bit.plist
Debian 7 64-bit.vmsd
Debian 7 64-bit.vmx*
Debian 7 64-bit.vmx.lck/
Debian 7 64-bit.vmxf
Virtual Disk-s001.vmdk
Virtual Disk-s002.vmdk
Virtual Disk-s003.vmdk
Virtual Disk-s004.vmdk
Virtual Disk-s005.vmdk
Virtual Disk-s006.vmdk
Virtual Disk-s007.vmdk
Virtual Disk-s008.vmdk
Virtual Disk-s009.vmdk
Virtual Disk-s010.vmdk
Virtual Disk-s011.vmdk
Virtual Disk.vmdk
Virtual Disk.vmdk.lck/
caches/
startMenu.plist
vmmcores-1.gz
vmmcores-2.gz
vmware-0.log
vmware-1.log
vmware-2.log
vmware.log

The disk images have the .vmdk file extension and in the above example have been split into 11 different files with numbered filenames. You don't need to defrag and shrink the numbered filenames, just the one without the numbers.

So in the above example, ignore Virtual Disk-s001.vmdk to Virtual Disk-s011.vmdk and run the command on Virtual Disk.vmdk.

If you do attempt to run the command on one of the numbered files, you'll get an error message like this one:

The file '/.../Virtual Disk-s001.vmdk' appears to be a sub-component of a virtual disk.
Did you mean '/.../Virtual Disk.vmdk'?
Failed to defragment: The specified file is not a virtual disk (0x3ebf).

Version used

I ran the above commands on VMWare Fusion 7.1.1 on OSX Yosemite 10.10.3.

Where are Vudu To Go downloaded movies stored on my Mac?Where are Vudu To Go downloaded movies stored on my Mac?

Posted April 19th, 2015 in Applications

The Vudu To Go app allows you to download movies you have purchased from Vudu* and stores them on your computer to watch using the app. But where are the files stored?

Location of Vudu To Go files on a Mac

The files are stored at /Users/<username>/Library/Preferences/com.vudu.air.Downloader\Local Store

When browsing for that location in Finder, click the "Go" file menu option and hold down the option key to show the "Library" option.

Then locate "Preferences"

Then "com.vudu.air.Downloader"

The "Local Store"

Under the local store you will find folders named with numbers. The media files are there.

Copying the files to another computer

It is possible to copy them to another computer in the same location. I attempted to copy the entire com.vudu.air.Downloader folder but it kept crashing on the other computer.

So I deleted it and ran the app again, started downloading the movie, shut down the app, copied the movie directory over and it worked fine. You probably don't need to go through the "start download and stop the app" step and can probably just copy the movie directory over, just probably don't mess with the files in the root level of that folder.

Can you control where they are saved?

No, but it might be possible to symlink the directory location to somewhere else, e.g. an external hard drive, but I haven't tried this myself.

Can you download rented movies too?

* I noted at the start that Vudu 2 Go lets you download movies you purchased with Vudu, but possibly it can let you download the ones you've rented to. I'm not sure about this, either way.

Tools for setting up and testing secure certificatesTools for setting up and testing secure certificates

Posted April 18th, 2015 in Linux/Unix/BSD

This is a quick reference page for some tools for configuring web server settings and testing secure certificates.

Mozilla SSL Configuration Generator

This tool provides you with SSL configurations for Apache, Nginx, HAProxy & AWS Elb, depending on the server version and openssl version.

https://mozilla.github.io/server-side-tls/ssl-config-generator/

To find out your openssl version, do this:

openssl version

To find out your Apache version, do this (depending on distro it'll be apache2 or httpd, and you may need to prefix it with /usr/sbin):

apache2 -v
OR
httpd -v

To find out your Nginx version, do this (again you might need to prefix with /usr/sbin):

nginx -v

Qualys SSL Labs SSL Server Test

This tool tests your secure certificate and gives it a grade. On a couple of older servers, I was able to improve the grade from an "F" to a "B" simply by using the Mozilla SSL configuration generator above. We should be able to get them to an "A" by fixing the intermediate certificate when we renew.

https://www.ssllabs.com/ssltest/

[Update] I bought a new certificate for a site using RapidSSL, the Mozilla configuration generator and with up to date Nginx & openssl on Debian 7 Wheezy and got an A+ with the testing tool.