[ampere]

Nice one there Kenney. Some questions:


- Would be interesting to know a high level picture of how you data mine (Dont need the specifics)?
- Can you take the two data and plot ISO-Odo/Year and ISO-Year/Odo charts?

[2000rpm]

Originally Posted by kenneyjacob

Wrote a small program to scrap data from OLX, it was a little tricky as they have some good counter measures set-up to prevent such attempts.

Since they have counter measures to prevent this, I assume this is something they wanna protect and keep confidential.

Counter measures = Virtual Locks in my understanding, hence breaking a lock can open you to law suits

I am not sure of the laws and the technicality, would be good if you can explain the approach you followed to get the data.

[kenneyjacob]

Originally Posted by ampere

- Would be interesting to know a high level picture of how you data mine (Dont need the specifics)?
- Can you take the two data and plot ISO-Odo/Year and ISO-Year/Odo charts?

There are many ways to mine data. You can use a tool called Selenium, which is testing tool primarily used to automate things we do on a browser. So it can browse pages exactly like humans do and save specific information we need from each page it goes through. More details can be found here - http://www.seleniumhq.org/

Or you can write your own code to extract information, which can be run on your local computer or a server. I used this option. For honda city the URL wil look something like this - http://www.olx.in/nf/search/honda%2Bcity

Second page - http://www.olx.in/nf/search/honda%2Bcity/-p-2
Third page - http://www.olx.in/nf/search/honda%2Bcity/-p-3

You can run a loop to get as many pages as you want. In each page the data is presented in a very specific format which is easy to scrap.

I did not understand your second question fully. Once the data is there we can plot any kind of graphs from it.

Originally Posted by 2000rpm

Since they have counter measures to prevent this, I assume this is something they wanna protect and keep confidential.
Counter measures = Virtual Locks in my understanding, hence breaking a lock can open you to law suits
I am not sure of the laws and the technicality, would be good if you can explain the approach you followed to get the data.

I dont think there is any law that says I cant save information that they send to my browser. The counter measures they use are small tricks which will look at unusual activity from an IP address and block it for some time. So they were looking at the frequency of page requests from an IP and blocking it if it exceeded a certain level which is not possible by human beings. We wrote our code so that it waits for some time before sending request for more pages.

Moreover I am not selling their information or publishing it somewhere, I am only using it to draw inferences. Maybe they should do this on their site, it will attract a lot of traffic.

[ampere]

Originally Posted by kenneyjacob

I did not understand your second question fully. Once the data is there we can plot any kind of graphs from it.

What I meant was :

- Plot curves for same year Price/Odo
- Plot curves for similar odo-bracket Price/Year

One gives an idea of reliability based re-sale while the other should provide idea on year based depreciation for the brand.

[kenneyjacob]

Originally Posted by ampere

- Plot curves for same year Price/Odo
- Plot curves for similar odo-bracket Price/Year

The following plots are for the honda city data. What can you infer from this ?

[ampere]

Originally Posted by kenneyjacob

The following plots are for the honda city data. What can you infer from this ?

On the first one, I can see that the cloud at least centered above 5L for the model. If we compare that across other cars, we should get a good idea about how others fare. As regards to the second one need to dig deeper.

[kenneyjacob]

Originally Posted by ampere

As regards to the second one need to dig deeper.

By looking at both these graphs is it safe to conclude that the age of the car has more impact on depreciation than the odometer reading ?

[ampere]

Originally Posted by kenneyjacob

By looking at both these graphs is it safe to conclude that the age of the car has more impact on depreciation than the odometer reading ?

Surely. But I think if you compare the same chart against other cars, you may see a different trend. (I mean the same graph). Again please note even I am thinking aloud here.

[alpha1]

Originally Posted by kenneyjacob

By looking at both these graphs is it safe to conclude that the age of the car has more impact on depreciation than the odometer reading ?

Also that the km reading can fluctuate the asking price ± 1 Lac from the mean.

Of course the variants (top end vs low end) would also contribute to this fluctuation.

[zenren]

Originally Posted by kenneyjacob

I dont think there is any law that says I cant save information that they send to my browser.

In this particular instance, you might be okay.

However, as a generic statement, it is incorrect. Unauthorized reproduction under copyright laws are even applicable in case you save content to your local machine that was intended only as viewable on their site. Classic example is the legal online streaming sites that show TV shows and movies (eg: Hulu) - Whether or not you intended to publish it later would be a very sensitive line that you might not be able to prove in case you land into some trouble.