Book review: Just After Sunset (Stephen King)

Author

Stephen King

Title

Just After Sunset

ISBN

9780340977163

Description

Collection of 13 short fictional stories.

Opinion

I’ve never really been interested in short stories but this book has changed my tastes when it comes to them.  Each of the stories in the book grabs you quickly and takes a good hold without feeling rushed or pushed to get too much story over quickly.  The pace is just right and the different styles of story-telling leaves each one feeling fresh and different from the last.  Out of all the stories, I only found one that didn’t catch my attention like the rest due to the slightly ponderous start, only to be saved by fantastic ending.

Several of the stories deal with mysterious happening without too much horror throughout.  I think most people who like fiction would find a few of the stories enjoyable.

Rating

I’ll give this book 9/10.  Really enjoyed it and gave me a new found taste for short stories.

CFTracker: Memory, Apps, Sessions

CFTracker is a CFC for ColdFusion (7/8) which provides methods for accessing the following information:

• Names of all active applications
• Get the application scope keys for any active application
• Get values from any application scope (CF8 Only) without changing the last accessed time
• Application details (time alive, last accessed, idle timeout, expiry status)
• Names of all sessions or per application sessions
• List of keys for any session scope
• Get values from any session (CF8 Only) without changing the last accessed time
• Session information (time alive, last accessed, idle timeout, expiry status)
• ColdFusion Memory information (free, allocated, free allocated, used, maximum allocatable)

Planned additions

There is a setMaxInactiveInterval method for application / session scopes but I’ve been unable to use it so far without triggering the last accessed time to update. Anyone with some Java reflection knowledge could probably work it out, *hint* *hint*.

Usage example

<cfscript>
  cfcTracker = CreateObject('component', 'tracker').init();
  apps = cfcTracker.getApplications();
  sess = cfcTracker.getSessions(apps[1]);
</cfscript>

Download and issue reporting

Thanks to the awesome RIAForge, you can download CFTracker from its project site.

Credit

• Ben Nadel
  For discovering the Java array.Contains() method
• Teeps (Terry Palmer) & his co-worker Java guy
  For getting getValueWIthoutChange working.

When not to use CfQueryParam

What!?  How could anyone say such a thing about the great CfQueryParam tag?

Well, for starters, I’m not saying don’t ever use it.  In fact you should use it as much as possible because it helps protect you from SQL injection attacks and improves the performance of your queries.  What I am saying is that in the situation I go into below, you really should consider avoiding it, but only if it’s safe to do so.  Plus the fact that I’ll pick on some other tags and stuff too but felt like CfQueryParam deserved to take the largest chunk of the blame.

Memory Leaks

My problem with CfQueryParam is about memory leaks.  While pushing ColdFusion 8 to accomplish the scenario below, the server crashed.  I couldn’t really tell what had happened so I added some logging and found that memory was being eaten up pretty quickly.  Now this is just one task, I’m not talking about several requests building up a memory leak over some time.  One task grabbing all the memory it can get until ColdFusion says it’s got no more and crashes.

Scenario

Given a CSV file, we need to read it line by line and perform several queries based on the data.  We also need to do this all in one transaction, since if there is any invalid data we’ll have to rollback everything we’ve changed so far.  The CSV file I used for testing had 68000 rows of data, each row having about 10 columns.

Now to parse the CSV I took advantage of some code by Ben Nadel that will convert it into a nice array.  He does note that if you’re going to hold that much data in one go, expect to use up a lot of memory.  So I changed things around a little.  Now I was reading in the file a line at a time using FileOpen and FileReadLine, resulting in a one dimensional array of the row columns to deal with and then discard.  This worked great, no memory leaks and extremely fast (thanks Ben ).

Then we add some queries and other things to the mix giving you this basic outline (not allowed to show you the actual code):

1. Create an instance of a CFC.
2. FileOpen the CSV
3. Loop until the end of the file
   1. Read a single line from the file
   2. Parse into an array.
   3. Pre-process and validate some of the data we’ll be using.
   4. Perform some queries
      1. SELECT some data
      2. SELECT some more
      3. INSERT a record
      4. UPDATE another
      5. EXEC a stored procedure
   5. Use the CFC to write another record to the database.
4. Another query to UPDATE all related records.

When things started to go wrong

Everything was written, I had a couple of test runs with smaller datasets (100 rows) to make sure everything was working correctly and I didn’t spot any issues.  When I threw 68000 rows at it I knew it’d take a while, but I didn’t expect the memory leak.  It made it to about 10000 records and died.

Rewrites and Memory / Speed graphs

There’s nothing like a graph to show memory leaking and here they are (I’ve limited them to 32700 iterations due to some logs being interrupted).  The blue line that hits the top of the chart first is the original version of the script.  You can click the image for a larger version.

The next graph shows the difference in speed between versions, not as important but nice to see.  The lower the line when it ends, the quicker the version was.

Versions

Now the graphs wouldn’t make much sense if I didn’t explain the changes.  Here is a description of each change and which item of the graph they are.

Version A

This is the initial template, still with its several CFQuery, CfQueryParam and CFC calls.  It made it just over 10,000 records before choking the server.

Version B

Commented out all the CFQuery tags (and therfore the CFQueryParams) so we’re only using the CFC to perform a query.  Made it further before crashing (18,600).

Version C

This really skipped a version but I threw again the CFC and combined the query it did with all the others.  Now we have a single CFQuery per row doing a lot of SQL.  It still only made it to about 25,000 until I removed the CFQueryParam tags as well.  Then it got up to 56,000, almost there but we’re still leaking memory all over the place on the way.

Version D

This version made it all the way to the end.  I decided to buffer the SQL so we only used one CFQuery for every 25 rows in the CSV.  Initially I used a string and it caused the memory usage to bounce all over the place, so I replaced that with an array being appended each iteration of the loop.  Then every 25 rows I’d ArrayToList(sqlBuffer, ‘ ‘) inside a CfQuery tag.  This was the fastest working version so far but it was still leaking even if it was at a slow enough rate to be successful.

Version E

I upped the buffer to 250 rows per CFQuery.  Which made little difference to speed or memory usage from Version D.

Version F

Those combined Queries that I was appending to the array seemed like a perfect excuse for a stored procedure.  So I packed up all the SQL and palmed it off on the Database (MSSQL) and ran it from there instead.  Initially I called this using CFStoredProc per row but this leaked all over the place.  I guess that CFProcParam might suffer from the same problem CFQueryParam does.  I went back to my buffer SQL (per 25) method but with an “EXEC sp_csvimport” each row, passing in the data as arguments (no cfqueryparam’s).

Not only did it complete the job, memory was STABLE! Looking at the logs I didn’t see a upwards trend, infact ColdFusion decided to garbage collect some of the memory back!

Version G

This was the same as Version F but instead of pre-processing the data in ColdFusion, I added some SQL to the stored procedure to do the job.  Memory usage was the same but it was a lot slower at the task.

Version H (The final one! I promise)

Finally I had reached my goal with Version F:

• No leakage trend evident in the data / graph.
• Fastest version so far.

The only change I made was the number of rows per CfQuery.  After playing around with different values I settled on every 100, which gave me the fastest so far and completed the job in 66 minutes for 68000 rows of data with no evidence of a memory leak.

Conclusion

Blimey, what a journey that was!  There are probably other ways around all this, like pre-processing the CSV itself and then using a tool to import it into the database then process the data using the stored procedure.  But I was up against a deadline and didn’t have time to rewrite the whole process.  Thanks to Ben Nadel for the CSV to Array code, that was a great help.  But the moral of the story is that even ColdFusion has it’s limits and certain actions can leak (CFQuery, CfQueryParam, CFStoredProc, CFProcParam and CFC calls) but are usually collected by the garabage routine.  It you go and do things on a big scale like this you have to work around those leaks otherwise you’re not going to make it.

So carry on using all those tags mentioned, especially CFQueryParam (security is important!), but if you do ever run into a similar problem on the same scale.  See what data you can trust and try and buffer those database calls.

Adobe CF8 SSLv3 Webservices

So you have some very sensitive data being made available via a web service which is secured using SSLv3 (Client certificates).  If you’re using ColdFusion 8, you’ll have to say goodbye to your good friend CfInvoke as he doesn’t speak the lingo (SSLv3).  Your old pal CfHttp does, but you’ll have to help him along by washing his mouth out with some SOAP.

In other words, CfInvoke cannot cope with client certificates and neither can his cousin CreateObject.  Strangely enough, CfHttp does support them but of course you have to do all the SOAP stuff yourself in order to communicate with the web service.

Here’s how it’s done

1. You’ll need to find out where the web service requests should actually be sent.  If you can get hold of the WSDL, open it up and look for something like this:

   <wsdl:service name="BobbyService">
       <wsdl:port binding="impl:bobby.cfcSoapBinding" name="bobby.cfc">
           <wsdlsoap:address location="https://localhost/bobby.cfc"/>
       </wsdl:port>
   </wsdl:service>

2. The location attribute is the one you’ll need, this can be different sometimes that the one you expect or get the WSDL from.
3. Now you’ll have to do some work yourself, figuring out exactly what the web service is expecting.  You’ll usually have some documentation to help you along, otherwise you’ll have to work through the WSDL and figure it out yourself.
4. We’ll need to built the SOAP request.  notice that foobar related to the method and the arguments are contained within it:

   <cfsavecontent variable="soap">
   <?xml version="1.0" encoding="UTF-8"?>
   <soapenv:Envelope
       xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
       xmlns:xsd="http://www.w3.org/2001/XMLSchema"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
     <soapenv:Body>
       <foobar>
         <foo xsi:type="xsd:string">Mister Dai</foo>
         <bar xsi:type="xsd:string">z</bar>
       </foobar>
     </soapenv:Body>
   </soapenv:Envelope>
   </cfsavecontent>
   <cfset soap = Trim(soap) />

5. Now we’ll use CfHttp to make the actual request:

   <cfhttp url="https://localhost/bobby.cfc" method="post"
       clientCert="#ExpandPath('./bob.pfx')#"
       clientPassword="*********">
     <cfhttpparam type="header" name="content-type" value="text/xml">
     <cfhttpparam type="header" name="content-length" value="#Len(soap)#" />
     <cfhttpparam type="header" name="charset" value="utf-8" />
     <cfhttpparam type="xml" name="message" value="#soap#" />
     <cfhttpparam type="header" name="SOAPAction" value="" />
   </cfhttp>

6. You result will be available via cfhttp.fileContent:

   <cfdump var="#XmlParse(cfhttp.fileContent)#" />

Conculsion

While this is probably a better workaround than having another server-side language act as a go-between, I still think Adobe should push support for client certificates into anything that’s making a HTTP request.  For example cfcache, cfschedule and the admin schedule task all make web requests which would fail if a certificate was required for access.  I also seem to remember that CfDocument makes requests for retrieving images.

Railo comparison

The score is now Railo 1 – Adobe 1 as Railo currently doesn’t support client certificates on it’s CfHttp tag or the others (CfInvoke, createobject etc…).  I’ve put in a request for it via their new UserVoice site, so please vote for it if you think you’ll ever need it.  Even if you only use Adobe’s engine since it’ll pressure them into adding it too

Railo 1 – Adobe 0

I’ve decided to keep track of my investigation into Railo after being brainwashed enlightened at SotR09 (London).  I’ll post up about any issues I run into, comparisons and features that I think puts it above or below the level of Adobe’s CFML engine.

Named arguments in scoped functions

This has never worked in Adobe’s CFML and is something that I always thought would be useful.  Especially considering that the same style syntax works perfectly well for CFC’s.

<cffunction name="foo" returntype="string" output="false">
 <cfargument name="bar" type="string" required="true" />
 <cfreturn 'Yes' />
</cffunction>

<cfscript>
 st = StructNew();
 st.foo = variables.foo;
</cfscript>

<cftry>
 <cfoutput>foo (no scope): #foo(bar = 'bob')#</cfoutput>
 <cfcatch type="any">No</cfcatch>
</cftry>
<br />
<cftry>
 <cfoutput>variables.foo: #variables.foo(bar = 'bob')#</cfoutput>
 <cfcatch type="any">No</cfcatch>
</cftry>
<br />
<cftry>
 <cfoutput>st.foo: #st.foo(bar = 'bob')#</cfoutput>
 <cfcatch type="any">No</cfcatch>
</cftry>
<br />
<cftry>
 <cfoutput>variables.st.foo: #variables.st.foo(bar = 'bob')#</cfoutput>
 <cfcatch type="any">No</cfcatch>
</cftry>

ColdFusion 7 + 8

foo (no scope): Yes
variables.foo: No
st.foo: No
variables.st.foo: No

Railo 3.1

foo (no scope): Yes
variables.foo: Yes
st.foo: Yes
variables.st.foo: Yes

Conclusion

I’ve filed this as a bug with Adobe and possibly even with Macromedia and they never acted on it.  But Railo have gone along with common sense and made sure it was possible from the start.  Support this syntax would allow you to, for example, store your own functions in the application scope and still retain the benefits of named arguments like optional ones.

Thanks Railo.

Sotch on the Road 2009 – London

I was one of the lucky ones who attended “Sotch on the Road 2009” in London.  I really enjoyed it, learned a lot and came back home with a head buzzing full of ideas.

I thought I’d briefly cover the sessions I attended so keep reading.

Keynote

After a little introduction by Big Mad Kev and Andy Allan (including a warning about chuggers), Adobe stepped in and gave a overview of where we are and where we’re going with ColdFusion and it’s related technologies (Flex / Flash).

ColdFusion 9 looks like its got the right sort of attention it needed when it came to new features.  ORM via Java’s Hibernate will make simple tasks with the database very easy and more maintainable.  More PDF functionality is available with methods for extracting images and text plus optimising the file size.

CFScript is being brought up to CFML level with its support and functionality.  I also thought that ColdFusion should had supported all functionality in both all along, glad it’s finally coming.

Oh and a multi-server management Adobe Air application.  Very fancy looking and could be extremely useful.

Bolt had what you’d already come to expect from CfEclipse but with the surprise of custom extensions being written in CFML.  Didn’t see that one coming and would certainly make it easier that having to learn Java.

Comparing Application Frameworks

Mark Drew took over from Sean Corfield who couldn’t attend.  He gave a good overview of the main frameworks available for ColdFusion and suggestions as to why they’re can be useful and organise your code.  Although he was a little lacking in knowledge about one of the frameworks listed, he did well to explain the others.

I came out of this talk with an urge to try out ColdBox as it has a great reputation for documentation and showed off some very interesting debug and cache information.

Advanced ORM in ColdFusion 9

Adobe delved deeper into the ORM that their packaging in CF9.  Adam Lehman gave a great talk and had some very good examples of the usage and what to expect.  A lot of it some looks too easy with very empty looking CFC’s that are automatically capable of covering the usual basic tasks of setting and getting data to and from the database.

He did, however, try and dodge several question about how stored procedures would work with this.  I felt he was either lacking in knowledge about if they could function via the ORM or knew that they wouldn’t and didn’t want to be negative.

Open Source Railo

Railo (apparently pronounced Ri-lo not Rai-low) is an alternative CFML engine to Adobe’s which is open source.  I’ve known about Railo for a while now but didn’t really feel it was near enough Adobe’s to be usable in my work place.  Thankfully Gert Franz brainwashed enlightened me on that.

Railo looks fantastic.  Everything that I need from a CFML engine is right there in the open source version and it’s compatiblity with the major CFML frameworks gives me a lot of confidence that it wouldn’t take much work to convert from Adobe CFML to Railo’s.

They showed off several small differences to Adobe’s offering that make life easier and squash those annoyances.  I think that sometimes Adobe / Macromedia would concentrate on the big new features and forget about the little things / obsticals that hamper developers in our day-to-day work.

On top of that they casually annouce that they’ll also be supporting ORM via Hibernate.  I thought that move was awesome, Adobe had done a great job of showing what ORM in CF9 is capable of.  So instead of repeating everything they let Adobe show the power of Hibernate and then just state that it’s coming to Railo too.  Genius

Securing ColdFusion applications

This session didn’t rate as highly as all the others.  The room was small, the display was wall mounted TV (not a huge one either) and the talk itself felt very dry.  I know security is probably a difficult subject to make exciting but I felt David Stockton could have done a better job at it.

Initially there was too much focus on his company and the security standards they adhere to.  Then some of the topics dragged on longer than they could have.  I’d have enjoyed it far more if he’d delved deeper into the common security mistakes, securing CF installations as much as possible and threat detection.

Although it probably didn’t help that this session was right after lunch and I was sat on the floor for it

Railo in the Cloud

Mark Drew strikes again.  Although externally hosted cloud computing is not an option for the organisation I work for, I still found this a very interesting talk.  If not for the fact that Mark gave away his password a few times due to plain text config files

I did come away with the impression that Railo is very capable at clustering and the extensions that are available make the job of keeping things in sync a lot easier.

Overall

Being the first conference I’ve ever attended, I wasn’t really sure what to expect.  The talks were fantastic and I felt I’d learned several things to take back to my job and vote for.  The seats were awful but I figured you’d rarely be able to get that many lazy boy chairs in time and for that price tag.  Finally, the food was nice but terribly organised but this was probably more down to the venue than the organisers.

Name / Job / Org badges would have been nice, plus I was very disappointed in the raffle at the end.  With that many developers hanging around, surely they could have knocked together some code to pick a name from the attendee list.  I’m just moaning because I wanted the Guitar Hero prize so badly

I don’t know if I’ll be allowed to attend next years, but I’ll certainly try!

Clan Web Application – “Clanster”

“Once upon a time there was a Dave who got bored and created a clan.  The clan needed a website and Dave with his little knowledge of web developer went out creating a home on the web for his clan.”

Well that was a long time ago and I haven’t even got a clan anymore, but I do have all these different attempts and versions of my clan web sites cluttering up my hard drives.  So I thought, why not review all my horrible old code and come up with some new horrible code.

Clan Web Application, or whatever it’ll end up being called, will be a package for clans to drop onto their PHP/MySql hosting provider to run their clan website, keep the clan history and help organise things.  It’ll probably use a PHP framework to make things easier for myself to write it, but hopefully not slow the site down too much.

Feel free to leave a comment if your intrested in lending a hand, sharing ideas, features, designs, a better name or anything else.  Maybe you even want to use it or need a clan web developer

Moving my virtual home

Well I got tired of my web host and the quality of their shared servers.  I know that you shouldn’t expect much from a service where you’re sharing with other sites but it was getting dire and they refused to do anything about it.  So I upped sticks and moved here.

Anyone who followed me here from my old blog might remember a post about writing a CakePhp blogging system for my own use, but that idea died with the web host.  Thanks 1and1, .

Hopefully I can start getting back into my blogging and maybe some people might find it useful or interesting.  I’ll be covering things I come up against during my web development work, plus general posts of life related subjects or a rant or two.

To anyone reading this, hope you stick around for at least a few posts