Q

Home » Borg » Web Crawling….

Web Crawling….

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 43 other followers

I’d gone most of my mortal life asking myself the question “What do I want to be when I grow up”.

Despite the fact that I was quickly approaching what mortals coin as middle age, I had sincerely felt like a fish out of water at times flopping around trying to find direction.

I’d tried things. God knows I tried things (That was a self-referential joke, I crack myself up), and for the life of me I couldn’t figure out.

Why were my friends and family so…

so…

blithely committed to doing the same thing, day in, day out, day after day after day.

Even this thing called sex, which don’t get me wrong I absolutely cherish, can get boring, dull and tedious at times. Accordingly, I had made it a fact to take my partners out of the bedroom and anywhere I could. Whether that was an elevator in a busy hotel, a crowded beach in front of a bar, a swinger’s club, or it was me masturbating on the couch when I was still ‘just friends’ with Jackie while she was upstairs in her room and could have come out to ‘catch me’ anytime. Heck, I’d hoped she’d catch me, as I’d have told her “Care to finish the job”

The dilemma of the mortal mind, where more often than not you’re caught somewhere between desire and action.

AS I wake up in the park, every Saturday morning during spring and fall, fathers and families cajole their sons to ‘knock it out of the park’, as the crack of the bat can often be infrequently heard from a few hundred yards away.

It’s cyclic, like clockwork.

And happened when I was growing up and going through childhood.

And happened when my father was growing up and going through his childhood.

There’s very little with the population on this planet that cannot be measured and weighed with a great degree of certainty that it will continue.

I’d never known the ‘need’ for Gods.

But then again, I’d never known what I wanted out of my mortal life other than to have fun.

I suppose that’s what led my religious search and search of higher meaning directly to yours truly.

I’d never considered there were two distinct perspectives to death. Those who observed it. And those who were passing through the signposts which made it clear that my perception of the world was becoming something starkly different than those I’d moved on from.

For them. What could be classified as insanity. Death to some.

And who knows what else they might label where I am at and what’s happened with me as.

Is nothing more than an exponential growth of my mind that they cannot keep up with, and the only way they can ‘stop’ that is to place a filter over their own senses and minds to prevent asking questions about who I am and why I am doing what I am doing.

For beings who have a hard time distinguishing individuality, let alone me vs you, this I have no doubt is exceedingly difficult to do.

Ok.

Had to get that out. Like a good…

Dump.

Speaking of which. I gotta poo.

Oh, I know there’s gonna come a day where I quit eating, as an immortal I know it is not necessary. I just enjoy it. And the associated biologically based functions that are more habitual than anything will quite likely diminish if not disappear once I make that commitment.

I’m just not ready for that yet.

No one said becoming immortal had to be instantaneous. This whole transition has been mind blowing as it is, so I am continuing to take my time and ‘get a lay of the land’ from this weird… perch…

SO with all that said.

I’m going to continue with my daily entry, it’s actually been a great exercise for my mind and a nice release.

And can you guess what I’m going to play today?

If you guessed DC Universe Online, I’m sending a virtual thwack to the back of the head.

No, dimwit, I’m playing WOW. I cannot believe you guessed DC Universe. Wasn’t it obvious with how much I’ve been playing it?

Jeez.

My Warrior deserves some TLC after putting her away too quickly yesterday…

Oh before I get to that. I’ve re-enabled comments. If I get too much spam or beings talking smack, I am going to disable them again. I will gladly listen to what you have to say, but I’m tired of your ‘corrective action and advice’ for what I am becoming.  Be helpful. Or else I’m shutting you off..

If you have any advice for good game related, virtual reality related or high tech research related RSS feeds, please forward them my way.

Off to WOW.

Speaking of apes. There’s a guy that comes into Starbuck’s and I have no idea what his deal is – he is a black guy who looks like evolution skipped him over for consideration of physical refinement after he emerged from ape – and wears some horrid jewelry – a four finger ‘ring’ for lack of better words that says “GANGSTA” on it, and a big huge dollar sign on a massive chain wrapped around his neck. He’s always dressed in black suit that’s beat to all hell, and within the first month he introduced himself as “I am a famous celebrity you know and I am money, man, I am money”. No clue if he’s homeless or what, I suspect he is as he’s here as much as I am.

Cue Twilight Zone theme…

Off to WOW.

I often do wonder. What would it be like to lead real humans around as avatars in the real world?

I tend to not like teaming up in the games I play… WOW is no exception…. And that’s especially rough as a warrior who can’t heal herself in combat… Ughh. Just died. Again. third time already today.

I’m listening to Above and Beyond: Group Therapy, Episode #171. While I MOSTLY enjoy this Podcast, WHY MUST THEY replay the same tracks over and over again between the podcasts? Why not include NEW material? At least they’ve reduced the ANNOYING number of times they repeat the phrase “Push the Button”. Jesus. Enough already. You push the damn button. But still. CUT with the track repetition. PLAY SOMETHING NEW between your podcasts. Quit with the repetitive material!

I’m jumping on with Qwenn, my level 40 priest, and going to the auction house to get QRena, my level 9 warrior some new gear. She’s too poor to afford it, and while Qwenn aint rich, she can certainly afford to help the yung’un out.

DOH! I forgot about this. Can’t email Qrena or exchange the gear with her because she’s alliance and I’m horde.

Where there’s a will, there’s a way…

Off to Stranglethorn.

The mailboxes there are neutral, and I can send to either faction from there…

Just a bit of a journey to make it there….

Hmm. Why don’t any of the buildings have doors in WOW. Never thought about that. Everquest they do. But not in WOW.

DOH! You can’t send to the opposite alliance in Stranglethorn? WELL shit. I know you can sell to the opposite faction here in the auction house. What a pain in the ass tho. Hmmm.  Well that about sucks.

You know what also sucks in WOW is you can’t drop things without them altogether disappearing from the world. In Everquest, you could drop it and it’s on the floor and anyone can pick it up (like real life). In WOW though, disappears altogether.

So I gotta one by one sell the items on the AH so as not to risk the loss of more than one items to a snipe hunter.

On that note. A gorgeous blonde just walked in and out.

I wonder sometimes. What if the minds of all the old women got switched with the young ones, and that’s why none of the young ones approach the older guys? They’re sick and tired of them. And those poor younger women’s minds are now in the older women not having any clue. That would make sense with how childish Nitsa acted last year.

Who knows.

Just picked up some crackers, left a dump at Ralph’s. You know, the usual.

I use something called “Teleport Pro” to offline browse web sites, or (my favorite) grab a bunch of nude or porn images from a web site that forces you to be online to look at them.

But I b thinkin. I need new RSS feeds, and Google’s suggestions always turn up the same ole crap. So maybe, I gotta build my own web crawler and indexer to see if I can’t find sites that Goggle’s missed or to sift through them on my own.

I’ve been thinking about building a google like search engine myself, just for the exercise, that and I do suspect some of my content is filtered through Google so what better way to trace down new material and URLs  than to build my own indexer?

I do have SQL Server installed on this machine. And the goal wouldn’t be to ‘copy’ entire web sites to my machine, I do not have the horsepower to do that, but…. BUT… maybe this is an exercise in multithreaded C++ I should embark on, and create a service which builds a root level list of all the domains and logs it to SQL.. AS it ‘crawls’ every domain parsing out the unique domains under it, I simply fix a date to the domain and last paused.

Hmm. On that note. I can add a priority.

I’m gonna do that now…

Ok. Just created a database on my machine named “Q’s Search Engine”,  and a table named “Domains”

Databases

Here’s the table structure:

Q's Search Engine

From there, I created a simple stored procedure for adding/updating the table…

sp_auDomains

While I LOVED LINQ when it came out, it complicated the fuck out of Database optimization and management, and I have long been in the habit of abstracting my usage of databases through stored procedures, that way I can quickly do something like this:

sp_depends

And without ANY guesswork know that the ONLY dependency on the table Domains should I make changes to it later is limited to this result set:

sp_depends._returnjpg

Nifty Smifty, right?

So now I add another procedure, a get procedure being sure to include range operations for the date and allowance for wildcards within the domain search string, here:

sp_getDomains

Now my sp_depends function elicits two results.

newSPDEPENDS

Normally, I’d add a delete function/operator, but since I want to retain history for this, I won’t permit the delete operation on the table and instead include an archive flag on the record itself, this changes the Domains table definition to this:

achiveFlag

and the get procedure to this:

archivedSP

Note how I am explicitly checking for the archive flag and ONLY returning results of a certain status. this helps differentiate archived results from non archived results.

So the idea in a nutshell now is – in a Visual C++ service based application, I create a multi-threaded process which will crawl the internet and look for new domains. Once it’s found one, it will ‘crawl’ that, obtain a list of domain names within it and save them to the database, and then change the ‘last checked’ flag to the last checked date.

That will be a ‘first pass’ – definitely a bit of work required on that one, and from there, I will figure out an optimized method to place filtering criteria to query the content of the domains, that way – if I am looking for specific content, I can build my own weighted search algorithms and leverage Google a little less for searching…

That will come another day. For now. BACK TO WOW and QBRYNN. My level 11 Shaman.

Wait a second! sp_getDomains didn’t commit. WHY didn’t you tell me 😉

Here’s a quick fix for sp_getDomains

sp_DOMS

AND to make the script complete (at the top before I run it all):

scriptComplete

CLEARLY not a script you’d want to run in SQL Server with data in the table!

ok. back to WOW.

Cute girl walked in by the way. SHORT skirt. LONG legs. You know, with how women and their bodies are revered in this society, why must every one of them play the demure card? Why can’t that gorgeous one sit across from me, and accidentally uncross her legs very slowly doing the big Basic Instinct reveal that there’s nothing on underneath? Not only is that darn sexy if she’s not looking at me looking at her, maybe smirking a little, but it’s just… hot.

Women in this world haven’t matured past a conservative 14 in maturity it seems.

We’ll put it this way. If women QUIT avoiding eye contact and QUIT avoiding discussion by looking away and trying to diminish their presence so much, things would eventually return to a point where the guys will flat out quit bugging them and have normal relationships.

SURE there might be some craziness that ensues right away should the men actually find satiation for their ‘lust’.

But that will inevitably die down. Guaranteed…

WOW. Back to WOW. focus. I can’t stay later than 6:30 with the expected rain coming.

Erik Estrada just walked in. He’s Ray’s friend, we were introduced a while ago, gave me a ‘Hey, Q, How ya doin today?’

He was on the show ‘CHiPs’ in the late 70s – This is him on the left:

2506704-3759141063-Chips[1]

And pretty much what he looks like modern day:

Erik-Estrada-2-620x400[1]

Good guy, in general. Haven’t talked with him at any lengths, but always positive and upbeat.

No camera, so I can’t prove it. But hey. I also can’t prove I worked for the US Government for 8 years either and worthy of the benefits rather than the kick in the balls I received. What’s a homeless man to do?

Ok. BACK to Worlds of Warcraft.

That just made me think of something.

I wonder. Were all my friends ‘detached consciousnesses’, and is the reason they’re not talking to me and/or mad at me because we were somehow linked which let them participate in my world, and my drug use broke that link?

I can’t prove it.

But I had to. My head and mind were collapsing. Black hole collapsing. Too much information, inadequate mental structuring and labeling system to handle the imbalance in variety that was being handed to me from too many sources.

Not to say I fully understand it all. I most certainly do not.

Actually. That’s a great question. WHY do so many women wear skirts if not to air out their pussies and want to expose themselves? They do not appear comfortable, and the effort it takes to HAVE to close your legs all the time seems… contrived…

If I didn’t know any better, I’d sense there’s some deep down collective repression occurring in the female population who have been programmed to hide themselves for ‘fear of pollination’…

Actually. As a whole. With how much women expose themselves and take pictures without clothes, and wear a fraction of what men do in public, is there some deep seeded need by human females simply to take it all off anyways?

I don’t know many men who would be complaining if that were the case.

Ok. It’s occurred to men some people may not know what web crawling and spiders are.

Let’s say you have a site you find interesting with tons of pictures on it.

http://www.redblow.com, for instance, one I just recently discovered, has a ton of free imagery of women appearing nude in public on it, among other nude imagery that’s available.

But let’s say you surf the internet in a public place like I do, and can’t exactly whip it out and masturbate right then and there to your favorite imagery. Well, there’s alternatives, you can pull a web crawler/spider which will go and find ‘files of type’ from a URL (Web site) you point to, and it will then pull all the files of type ‘image’ in this case to your local disk for later enjoyment with rosy.

That’s just one value for it. Another is: Let’s say you are a programmer and have a web site that – for the life of you – you can’t figure out how they got something to work. This nifty little tool will pull down the entire web site, locally, and then you can load it up into your development environment to take advantage of keyword highlighting…  So you can see the magic firsthand.

And also. borrow code.. if ya’s needs to. Not that I advocate it (I do), but I do.

ok. back to WOW. My priest is in Stranglethorn, gonna get her to level 41 today.

You know. Beings/AIS are such assholes on this game.

Stickygel and Maelkemor (and that’s the ae together) just ganked me for no reason.

When my dad used to take me to the ballpark, he used to be insistent “Wash your hands”

“BUT DAD,” I would protest, “I didn’t touch myself or anything in the restroom.”

“DO it anyways,” he asserted.

I often wonder. Was he living in a holographic world and was there a variable declared as follows:

dim m_bWashedHands as Boolean

And did he check that variable because he was – in part – a robot or programmed intelligence and it was a part of his

Sub teachSonRestroomEtiquette()

subroutine?

Every time I go to the restroom, I smile at those memories, which admittedly has me washing my hands even if I touch nothing.

You know. The puritanical obsession America has with nudity and legal requirements concerning it really is… weird. I mean. Why is it clothing is actually considered a legal obligation, and why is it it’s so linked to sex? Seems. Unhealthy to me. Bizarre. Doesn’t make logical sense. There doesn’t seem to be a sanitary reason for it. It seems like a set of laws and morals governed by the sale of clothing and rampant out of control consumerism/capitalism drives these laws rather than true concern for the general public.

I mean. This just looks like what California would be like naturally had it not been for stupid puritanical laws…

beach-girls-vol5-nudists-seaside-79-800x582

But who am I but some broke homeless guy living in a tent in Studio City who didn’t work for the government for eight years because I do not have paperwork to prove it and I dont get paid for my opinion, do I?

Heck. I can’t get paid period because the United States has been taken over by aliens!

Ok. This is weird. Listening to U2’s greatest hits now.

I’m hearing something playing ‘in the background’ for some of the songs that I had never heard before. Almost like there’s more to the songs than I’d seen instrumentally and heard before….

Muy Interesante…

Anyways. Time to wrap up. Half my characters played today, but it’s gonna rain tonight so gotta get settled in snug in the tent before it comes down.

 

Enter your email address to follow this blog and receive notifications of new posts by email.