What happens when your app hits the frontpage of reddit

Never thought I’d see this

TL;DR version:

  • You’re probably going to get 1,000,000+ unique visitors.
  • Your site will get a lot of media coverage.

Here are quick and important tips:

  • You should implement a CDN like cloud flare will help prevent the hug of death.
  • Definitely make your site mobile compatible.
  • Define and implement your call to action before launching.
  • Get ready to take action; you only have a 48 hour window where the majority of traffic will come from.

The Story

I knew I built something kinda clever and kinda cool at the end of the 48-hour hackathon, but I didn’t think it was anything too great.  “A 60-second meditation tool for stress relief” — overall, people seemed to like pixelthoughts.co.  I wanted to fix up a few bugs before putting it on reddit, but I was so busy with other projects, it was shelved.

Reddit — where all things go viral

Fast forward a month, I dusted off pixelthoughts, slapped on a few bug fixes, registered the pixelthoughts.co domain, and created a cloudflare account just in case it got popular.

I uploaded pixelthoughts.co to /r/internetisbeautiful and thought it might get a hit thousand hits.

I refreshed the page every few minutes, and quickly saw that there was no movement.  In fact, /r/internetisbeautiful had given it 2 downvotes, and 1 upvote.

‘Ah well, there’s always the next project’ and left for lunch to grab some indian food.  Returning about 45 minutes later, I checked the status and saw that I suddenly had 21 upvotes and 3 downvotes.  5 minutes went by and the post now had 28 upvotes and 3 downvotes.  Pixelthoughts.co was now on the ‘hot’ page on /r/internetisbeautiful – I thought ‘well, this is going a lot better than anticipated’.

Little did I know that 2 hours later, I’d be on the front page of reddit with over 7700+ concurrent connections.

Realtime

Woah… that’s a lot of concurrent connections

Continue reading

Creating a recursive web scraper with Node Crawler

So you want to scrape lots of data from the internet using Node Crawler? No problem. You’ll need to know JavaScript, jQuery, and a bit of Node.

First things first:

1) goto Terminal and create a new javascript folder called node crawler and save a file called “craigslist.js”
2) in your folder, run npm install crawler
3) Make sure you know how to select elements from the DOM using jQuery
4) Pick a site you want to scrape from. In our example, I’m using craigslist.

Lets jump into it and I’ll show you the code in full and then break it down piece by piece:

when you run node craigslist.js This will give you a result such as this:

Beautiful bike data

Awesome, lets break down how it works:

Lines 1 -3: All of our require statements. Only module you need to require is crawler.

Lines 7-9: instantiate a new Crawler and pass in standard options. on line 9, create your callback function.

line 11: This will loop over every instance of .row class on the page.  As you can see, the row class contains all the pertinent info we need.
$('.row').each(function(index, a) {

blogpost

lines 14-18: using jQuery, select data you want to save and put that data into the json object we’ve created

We want to append that data to the JSON array, and when all is complete — we want to find the next page.
Go recursion go!

Line 26: Here we look at the an item on the on the HTML document that we can pass into our crawler and find the the next page. Thankfully, craigslist gives us a range element that we can pass into the URL to find the next 100 elements. We extract that of course.
var rangeNumber = $($('.range')[0]).text().split(' ')[2]

Line 30: Create a ‘link’ to pass into the crawler based on the range we just scraped
var toQueueUrl = 'http://sfbay.craigslist.org/search/bia?s=' + rangeNumber

Lines 32-38: We want our base case — which is “If the range is less than 1000, run the next link into our crawler. Otherwise, save all the data to our system.”

Line 43: Start the recursion and pass in the first page
craigslist.queue('http://sfbay.craigslist.org/search/bia')

I partied with my web browsers!

I just moved to SF just over a two weeks ago, and don’t have any friends! Oh so lonely — who will party with me? Well, since I’ve been hanging out a lot with my web browsers recently, why not invite them to party!

partyMarc

hmmm, who to invite?

I wrote a little script with recursion, and bam instant party!

Click here to party: 

Drag this Link to your bookmark bar for future party use! It will turn ANY SITE into a party!

Continue reading

I launched my first web business!

So I’ve finally launched http://www.kindreminders.com

There was so much buildup to the launch. So many “what if” questions running through my mind. My lizard brain was running full capacity:

  •  What if I get 100,000 hits?
  •  What if I get too many orders?
  •  What if I get no orders?
  •  What if people love the site?
  •  What if people hate it?
  •  What if a bigger company comes along and steals my idea?
  •  What if I’m a failure?

Continue reading

Soundcloud

So I just joined up with SoundCloud.

Decent platform to share music creation online.  Overall, I’m happy with it.  Way back in the day, I would use “newgrounds” and “redfizz” to share music — how times have changed.

Love to see so many online music portals they have nowadays for emerging artists.  More freedom for peer to peer!