SQL Server not starting after an update

So this morning after the patch weekend following Microsofts Patch Tuesday I updated our SQL 2008 R2 server with a security patch (KB2494088) and rebooting the MSSQL Server service refused to work. It would start up but when trying to login all you got in the SSMS(SQL Studio Management Studio) was "login failed for user. Reason: Server is in script upgrade mode. Only administrator can connect at this time"



And after 2 minutes the SQL service would stop. And you're back to square 1. In the event log all i was seeing was the following two errors:

Script level upgrade for database 'master' failed because upgrade step 'sqlagent100_msdb_upgrade.sql' encountered error 598, state 1, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion."
and


Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.

This is where i sh*t myself! After a quick trip i composed myself and decided to try a rebuild of the database. In SQL 2005 and SQL 2008 R2 i know its different that SQL 2000. To do a rebuild in SQL Server 2008 R2 i used the installation media ( you can use the local setup file which lives in C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\SQLServer2008R2) and used the setup.exe file and ran the following command:

SETUP.EXE /ACTION=REBUILDDATABASE /INSTANCENAME=mssqlserver /SQLSYSADMINACCOUNTS="DOMAIN\administrator" "NT AUTHORITY\SYSTEM" "NT SERVICE\MSSQLSERVER" /SAPWD=sapassword

This ran successfuly (it didnt come up errors once it has run) but alas the same problem, so rebuilding the master database didnt work.

What did work for me was stopping the MSSQL service and starting it up again by using the below commands:

>cd "C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\"

>sqlservr.exe -c -m -s SQLinstance

>sqlcmd -S SERVERNAME,1433 -E

 

Now the switches on sqlserver allow it run not as a service (-c) and in single user mode (-m) and the -s allows you to specifiy the server name. Running SQL in single user mode means no-one can login normally and allows SQL to get on with its upgrade script in peace.

I left this to run for around 15 minutes (if its a busy SQL server you'll see many attempts of other servers/workstations logging in. To stop this i just pulled the network plug as I was working locally. If you need to remote on don't do tis...for obvious reasons :p) and after this finished and I can could see the upgrade script ran successfully I was able to start the SQL service up in the Services.msc and login OK using the SSMS. No need to re-run the patch or roll back and get screamed at by any developers (which could of been the case for me)

After this fault happened I now always allow SQL to start in single user mode after an update/upgrade so it can run these scripts peacefully. Its not really needed and its mainly me siding on the side of caution but it never hurts :)