Informed Networker Blog

Today I (Dave Mackey) am going to briefly jump off-topic and talk about the economy. WaMu has just sunk due to a “run on the bank” and its assets/liabilities sold off to J.P. Morgan. This follows a series of significant and stunning crashes - Countrywide, Bear Stearns, Fannie Mae, Freddie Mac, Lehman Brothers, Merrill Lynch, AIG. Companies that are household names - and underlie the financial cornerstone of our nation.

Lets begin by talking about my qualifications to discuss this economic crisis…Okay, glad that is done. Seriously though, I am not an economic expert, and perhaps some “experts” can throw light on why everything I’m saying is wrong. My knowledge of economics comes from a high school course and personal entrepreneurship. My knowledge of the current economic crisis comes from daily online news readings, watching the Colbert Report and Jon Stewart (yes, they are comedians), and reading articles from my weekly subscription to Newsweek. All that to say, I come to this debate with only what I would consider the fundamentals of logic - rather than any understanding of the magic complexity behind our financial system.

There are so many topics we could discuss. For example, “Whose fault is this economic crisis?” “How could this crisis have been prevented?” or “What will be the effect on web 2.0 companies?” But I’d like to look at just one particular topic in general: “How should the average American respond to this crisis?” This advice comes with no warranties - utilize it at your own risk.

Do not fear. Why? Isn’t there something to be afraid of? Most certainly. These are uncertain times in which corporations we trusted to act responsibly have failed to do be responsible and our government does not inspire us with confidence. The question, however, is not whether there is something to be afraid of but what our fear is gaining us. Quite simply - our fear gains us a tighter downward spiral towards economic depression. Why did WaMu go belly up? While we can point the finger at mismanagement the immediate cause was a classic run on the bank. People became afraid and withdrew large amounts of money from the financial system in a short period of time.

So what do we do? I for one am going about life as normal. Do I know that I will have finances in the future to continue in this matter? No. But do I know that flipping out will only make matters worse? Yes.

Lets think about the positive results of our fear. What do we get out of withdrawing all our money? Stopping all our spending? Nothing. “But wait, if I withdraw my savings, I have money…and it can carry me through!” Hmmm…

• Do you have enough? Most average Americans don’t have enough finances to last more than a few months. This is guaranteed by the FDIC (up to $100k) and most likely if we were to enter a severe depression it would last much longer than our measly savings could carry us.
• How much will it be worth? We can expect the U.S. Government to print money to keep fluidity in the markets. Ever hear about what happens when governments do that? Money loses its value rather quickly. So, by withdrawing we are taking a gamble - we are hoping that things won’t get bad enough that the government will print money (and our money will lose value) while at the same time we are helping the economy become worse.

The articles I am reading - the things I am seeing on the news - are all talking about the importance of “restoring confidence.” What if we simply chose not to lose confidence? I’m not saying longer-term we shouldn’t be smacking around some of these companies and individuals who brought us to this brink, I’m just saying that the quickest way to come out of this economic crisis (IMHO) is to drop the fear.

Now, if everyone would tell me why I’m wrong…I’d appreciate it.

We are still ironing out all of the fine wrinkles, but think everyone should be pleased with two new improvements we’ve rolled out:

• Web Server Load Balancing - We now have two www servers that share the load (distributed to them via a dedicated load balancer) between them. This means if one server fails, the site can still remain up. It also means we are able to handle twice the load. Remember, our web servers are only 256 MB VPS - as we prefer to scale sideways than depth (and can always increase depth later).
• Database Server Beefiness - We’ve doubled the memory, CPU, and disk capacity of our database server. We are working on implementing a cluster of db’s, but right now the honest truth is that one DB is performing fine and putting up additional db’s would be a significant expense (b/c cluster can’t run on 256 MB).

Additionally, we’ve begun running webYog’s MONyog, a SQL analysis tool which has been giving us some great pointers when it comes to analyzing and correcting slow queries/configurations.

All of this is in preparation for the mTurk integration and a major push forward. The date will be soon arriving and we cannot wait for the opportunity to show off the new features of IN!

The beta of our mTurk integration went through a first round of testing last night - and held up fairly well. There is still some significant work to be completed, but we are aiming for a full release within the next week.

For those who don’t know or have forgotten, integrating with mTurk will allow IN to utilize a massive, global workforce to help edit and summarize news stories as they happen. This will increase the depth of our coverage and allow our in-house editors to shift their attention to quality control and higher-level taxonomy.

As I mentioned in a previous post, we are currently in a “calm before the storm” period here at Informed Networker. While things seem to be staying at a steady pace, behind the scenes we are revolutionizing the way we work to launch out in a whirlwind of activity and growth. Expect this storm to hit within the next 1-2 weeks!

In the meantime, stop over at the SWCMS blog. Its a good way to keep track of IN’s progress as well, as SWCMS is the underlying code base (without our proprietary developments) used by IN. You’ll see we have been making great progress and released a long-term roadmap that reveals some of the features you can expect to see integrated into Informed Nertworker in the near future.

I create lists - lots of lists. Usually I create these in a point/sub-point/sub-sub-point fashion. Whether for my personal life or determining future roadmaps these lists are of tremendous value to me. I’ve run into an issue, however, which I have not the time (nor ability) to rectify…Yet which I believe would be of significant value to many.

Essentially, I need a way to make collapsible/expandable content within a traditional office word processing document (whether Microsoft Office, OpenOffice.org, or Google Docs). See, sometimes I just need to see the names of different “projects” or “headings” whereas at other junctures I want to drive down and view the details…Yes, I could accomplish this by writing HTML or by using hyperlinks within traditional documents - but it would be so much simpler if I could just write it all in one document and then expand/collapse the details on an as-needed basis. Thoughts?

I don’t do much of the development work for Informed Networker. There are lots of other tasks to do, and my specialty is not PHP/MySQL, my background actually being as a network engineer and hobby wise I wrote in ASP.NET/MSSQL. That said, I still do enjoy developing - for IN and on the side. There is one area that frustrates me significantly, however. I don’t have time to commit to developing a solution, but think this is a worthwhile project and would love to see it as an add-on to some popular IDE’s (e.g. Visual Studio, Zend Studio, phpDesigner).

Essentially, commenting is important in development. Its important so you know why you did what you did, but its even more important when someone else has to maintain the code you wrote. Unfortunately, commenting as it should be done can be extensive and annoying. Scroll through five lines of comments for two lines of code. What a pain. Okay, its not always that way - but it can be.

I’d love to see IDE’s offer collapsible comments - or perhaps even remove the comments entirely and just show a small Info button that could be clicked to display comments. This would allow developers to be as verbose as necessary without crowding their code. Thoughts?

Things have been a little quieter around here than normal, but I’d like to suggest that it is the “quiet before the storm.” We are working as fast as we can on several fronts:

• First, we continue to post content on a daily basis. We want to keep the site alive, even at the same time that we recognize that our editorial staff cannot handle the volume of stories currently appearing.
• Secondly, we are working to automate much of the editorial process using Amazon’s mTurk. Our developer has already committed sixteen hours to this task alone…and will continue to rise to the occasion and release in the near future an amazing mTurk integration.
• Finally, we have been working closely with SWCMS. Dave is one of the core team members of SWCMS - a fork of the popular Pligg social news project that IN is based on. This provides IN significant advantages as we are able to work in an intimate fashion with the core developers of the SWCMS future (and we are happy to announce the 1.0 release of SWCMS is around one month away).

So, I’ll keep writing and we’ll keep posting stories…but know that even amongst the hubbub bigger things are right on the horizon!

Here’s a little insight into what we did today:

• Continued, as usual, submitting numerous stories to IN and commenting upon stories as appropriate.
• Testing a trial version of webYog’s monYog which monitors MySQL databases for statistical anomalies indicating problems in performance or security.
• Dave is training himself more in-depth on Google’s Ad Manager using the video tutorials and documentation.
• Work on developing mTurk integration is continuing. The developer has familiarized himself with sample code and the api and is preparing for actual project implementation.
• Offered up a beta of our new Akismet anti-spam module for SWCMS/Pligg.
• Ongoing coordination with the SWCMS team.

In my last post I discussed the great challenge IN is currently facing - so many stories even in our core bundle that its overwhelming for the editorial staff (though they have been amazing!). With this I mentioned that one of the ways we intend to counter this problem is by integrating with mTurk. Development on this project will begin tonight! For those who are interested I’ve created a rough diagram of our current workflow with mTurk integrated into the mix. My goal is to have this functionality operating within 5 business days - so expect to see this functionality sometime late next week.

Additionally, we’ve hired a developer full-time for the month of September. Previously he had been working for us 2-4 hours a day, several days a week - now we have him for a full eight every day. We hope to renew his contract long-term at the end of this month, meaning we will have two full-time employees - myself and the developer.

Well, without further ado, here is the workflow diagram:

P.S. Thanks to Gliffy for their excellent web 2.0 diagramming application alternative to the expensive Microsoft Office Visio.

So here is the great challenge Informed Networker (IN) currently faces. There are many, but this by far is the largest: Quite simply, the body of articles concerning IT is larger than we could possibly have imagined.

IN uses an RSS Importer to grab stories from several dozen RSS feeds which we believe represent the core IT news - TechCrunch, GigaOM, Ars Technica, The Register, and the various ZDNet Bloggers. Our analysis shows that stories from these sources alone average well over 200 stories each day¹. That is a heck of a lot of stories, and we have been unable to keep pace.

So how will IN respond to this challenge? I’m so glad you asked. Its not by giving up. Rather we are:

• Working towards integration with Amazon’s mTurk. This will allow us to shift excess editorial work onto a globally distributed and scalable workforce. As this occurs editors will focus from editing the actual RSS stories to reviewing the content edited via mTurk.
• Working towards opening editorial controls to the wider user base. Soon user’s will be able to see these “unedited” imported stories and offer their own edits of the story. Editors will approve/deny edits. Further, using the points system we will give incremental rights to wider users - preventing spammers from editing massive numbers of stories²
• We also intend to shift some responsibility to webmasters. While we have hit a lot of the core blogs, there are scores of other blogs we’d love to follow but don’t have the time. So, we’ll be working one-on-one with bloggers and giving them a custom control panel which allows them to import their blog, edit/post their articles, all in a streamlined fashion. Yes, there is an approval process built into this as well - so only the select, the few, will be able to use these controls - we aren’t opening ourselves up to every blog - nor even the majority of blogs.³
• I’ve moved full-time as of this past Thursday. Starting tomorrow will be my first full-time workweek at IN. Expect to see a dramatic uptick in the number of articles I personally edit.
• We are moving towards establishing a long-term, full-time relationship with our current outsourced/part-time developer, increasing the speed at which we can develop new features.

What This All Boils Down To…

For the time being we will continue to offer a great source for finding and discussing IT news and articles. Over the next several weeks expect to see us increase our ability to handle the massive amount of news many hundred fold as we dynamically scale our systems and workforce.

At the same time, we will continue to work on innovations for IN in other arenas - e.g. the visual story relations will continue at a similar priority level to the above.

Every challenge is an opportunity, not an obstacle. We adapt and change at the speed of light to the needs apparent…hold onto your seats folks, this will be a great ride.