tarpontech
05-18-2010, 05:43 PM
Using VieleRETS to automatically update WP-Realty
What this covers:
1. How to configure VieleRETS for use with WP-Realty
2. How to configure a batch job in VieleRETS
3. How to configure a batch job to run in CRON
What this does not cover:
1. How to install VieleRETS and perform general configuration (i.e. communicating with your RETS server)
2. How to install or configure WP-Realty
For general VieleRETS setup information,Trent Richardson (http://trentrichardson.com/2008/02/06/rets-retrieval-with-php-and-vielerets-part-1/) created a terrific blog postthat I found invaluable when starting out.
Creating Sources: Telling VieleRETS what to download, and planning ahead for WP-Realty integration.
Setting up your initial sources can be…problematic if things don’t go well out of the box, but between Trent Richardson’s blog posts and the VieleRETS message board, hopefully you’ve been able to get communication up and running between your RETS server and VieleRETS. We’re going to focus on issues that directly pertain to getting VieleRETS and WP-Realty to play nice.
The first thing to know is that you will need to create a Source for each of the property types you wish to display listings for, examples being Residential, Commerical, and Lots (land). Make sure to name your Sources per Property type, don’t just call them Default, Default_1. Name your Sources Commercial, Residential…ect…you’ll thank me later.
You’ll notice that part of the process of creating a source is choosing which fields to download. It’s entirely up to you what fields you wish to use and/or display, but take note of what you pick. You’ll need to create the listing fields within WP-Realty that correspond to the fields you are downloading from VieleRETS.
This step is VERY, VERY important. Take your time, think it out, consult with your clients/agents/friends/family, whatever, but don’t take this step lightly. Save yourself A LOT of additional time and work, and make certain you’ve selected every RETS listing field you may, possibly, ever need, and make certain that you’ve created a corresponding listing field inside WP-Realty.
Missing a field at this stage will cause you to have to start all over again if you decide you really needed that “other rooms” field a month from now. You’ll need to purge your database, recreate all of your VieleRETS configuration, and then download the whole database again.
The fields inside WP-Realty don’t need to match exactly what they’re called in RETS, such as, the field in RETS is called Subdivision, I called it Neighborhood in WP-Realty, but it’s recommended to keep things the same. It makes things a lot easier when you are sorting through 120 fields to match them up. You’ll see why when we get to creating Extracts.
Lastly, while I can’t help with the myriad different ways VieleRETS talks to different RETS servers, I will give a little advice. RETS servers like for folks that are querying it to ask for certain minimum information. In my case, most servers like you to at least query for City, ListPrice, and Status. If you can’t seem to download anything, I’d check to see what fields you are using to Query, first.
Ok, so, we’ve got Sources set up to match all the Property Types we’ll be using, and we’ve selected all the fields we’re going to download for each Source, and we’ve created all the corresponding fields inside WP-Realty.
Creating a TARGET: Telling VieleRETS how to find WP-Realty.
VieleRETS needs the full root path to your wp-realty plugin installation. How to determine this depends on your hosting environment. Most shared web hosting accounts stuff all of your files inside of your account’s /home/ directory, but if you are unsure of your root path, save yourself much time and aggravation and call your hosting tech support and get the right answer.
Once you’ve determined the root path to your wp-realty plugin installation, you have two possible ways of using VieleRETS’ menu to navigate to it.
If the server’s permission structure allows, you can just click the radio buttons corresponding to the correct folder you need to navigate into, or, more reliably, just type the full root path into the field labeled
“Full path to Open-Realty install”.
It should look something like:
/home/your_hosting_account/public_html/wp-content/plugins/wp-realty
One little trick about VieleRETS is that it may not automatically recognize the path once you’ve typed it in.
By clicking someplace else on the page, such as clicking into the Name field at the top (just don’t change anything!), VieleRETS usually refreshes and will navigate to the path you’ve specified. When this happens, the screen will change to display the list of WP-Realty fields available to be populated by MLS data.
If you don’t see the screen change to show these fields, you most likely have a file permission or php permissions issue, both of which are outside the scope of this document.
Let’s hope you didn’t have a permissions issue. I’ve used this system on 4 different hosting accounts, with different results, but when permissions and PHP configuration are correct, creating a Target is dead easy.
Creating an Extract: Getting VieleRETS and WP-Realty to play nice.
Once you’ve created your individual Sources within VieleRETS, you’ll need to set up your Property Types within WP-Realty, using the EXACT names being used by VieleRETS. Doing this while creating your Extracts is a good idea, as you’ll have the Property Type names (called Class Name Style in VieleRETS) staring you in the face during the Extract creation process.
Word of advice: If VieleRETS is using Residential, don’t create Property Type residential in WP-Realty, it’s case sensitive.
Once you’ve established what property types you’re going to use, take note of the Property Type names in use by your MLS. My best example of this is in regards to Multi-Family units such as condos. Many RETS servers will provide you with the option to use Condos/Townhomes or CommonInterest as possible property type names. The WP-Realty database does not play well with Property Type names that use a slash, such as Condos/Townhomes, so you’ll want to choose CommonInterest, or another Property Type name that doesn’t use a slash. Remember, Case Sensitive.
Creating an Extract isn’t difficult, but it is tedious, which is why its so important to have all your fields set up correctly in the Source, and in WP-Realty, not to mention having everything named right.
To start, name your extract per Property type, as well, you’ll need one extract per property type. Each Extract reads from its own Source, but luckily they all point to the same, shared Target (wp-realty’s database).
You’ll need to choose the corresponding Source, (see why naming is important?), I’d take a moment to consider the “maximum images to download” option. Something around 10 images is probably more than enough to displaying a property. Some overzealous realtors can stuff a lot of images on the MLS for a single home, and your server’s storage capacity and bandwidth limitations won’t be so kind to you.
The last setting on the main Create an Extract page is the Class Name Style. We’ve already gone over this at the beginning, but be sure to pick a class name (property type name) that plays well with the WP-Realty database.
Once done with this page you can apply your changes, and you’ll be confronted with a menu of options.
Now is the time for some tedium. The only other area we need to focus on for an Extract is the Data Map. Don’t mess with the (Expert) data map. The Data Map page is self explanatory. Here you tell Viele-Rets where inside the WP-Realty database to put the data it is downloading. If you followed my advice and named your WP-Realty fields the same as the RETS fields, it’s a simple matter of matching names. Once done, triple check, however. You don’t want prices to start showing up where half-baths are supposed to display.
That’s really it for creating an extract. Now you just have to do it a few more times for each Property Type. Enjoy.
The moment of truth
Now we find out if we have any idea of what the hell we’re doing. Time to run an extract.
From the VieleRETS main menu, choose Download Listings.
Choose an extract and Apply.
Your download Criteria are dictated by the “Fields to Query” you set up during Source Creation.
If you don’t get any results, you may have to fiddle with these fields till you get something your RETS server likes.
In this test case, I’m using Status, City, and ListPrice. You may want to “Set Download Limit” to 10 or 15 listings to start, we’re just trying to make sure the system works, we can pull the entire contents of the MLS after we know things are a go.
The last thing I want you to take note of before we apply our settings to run an extract is the “Update the Batch Control File” Later on we’ll be talking about automated batch jobs, and using this feature can save us a little time in creating our batches. For now, leave the box checked.
If things went well, the batch will complete, and you’ll see a brief summary displayed at the bottom of the page detailing how many listings and images were downloaded ect…
If you didn’t get a successful extract, but you’ve gotten this far, you probably (hopefully) just need to tweak your “Fields to Query” in your Sources, and adjust what criteria you use when running the extract.
For those of you that got a successful batch, you should be able to view the listings inside your Wordpress blog, depending on how you’ve set up WP-Realty. Or, you can view the listings under the “Edit My listings” or “Edit All Listings” inside the WP-Realty Control Panel.
If you’ve got properties, you’re in business.
Creating Viele-RETS batch jobs and running them automatically via CRON
I’m skipping ahead a little bit, and I’m not going to go over how to run a batch from the command line, the VieleRETS message board and Documentation goes over that well enough. What we’re going to talk about is how to fill up your wordpress blog with every single listing your RETS server will allow you to download.
It’s actually really easy, just takes some busy work. First, lets look at a sample batch job:
<VIELE_BATCH>
<EXTRACT> residential
</EXTRACT>
<!--<QUIET/>-->
<UPDATE_BCF/>
<!--<UPDATE_ONLY/>-->
<LIMIT> NONE</LIMIT>
<QUERY_VALUE name="ListPrice">400000+
</QUERY_VALUE>
<QUERY_VALUE name="ListingStatus">|A
</QUERY_VALUE>
<QUERY_VALUE name="ListingID">
</QUERY_VALUE>
<QUERY_VALUE name="ModificationTimestamp">
</QUERY_VALUE>
<QUERY_VALUE name="City">|Watoosee
</QUERY_VALUE>
<LAST_RUN>2010-04-20T17:28:33</LAST_RUN>
</VIELE_BATCH>
If you stare at it long enough, you’ll see that by running this batch job I’ll be running an extract on the Residential property type for Active listings over $400,000 in the city of Watoosee.
The basic premise is that you’ll have to create a batch job file for every query you wish to run.
In my particular RETS, I have 3 price categories, -$300,000, $300,000-$400,000, and $400,000+.
If I want to download all active Residential listings in Watoosee, I have to create 3 batch jobs, one for each price range. Want Commercial Properties in Watoosee? That’s three more files. Yes, you’ll be making lots of batch files. Not only making them, but please name the files something intelligent, as well. You’ll need them named appropriately for adding to your cron script, and for logging and troubleshooting.
You can get started by taking this batch file, however, I recommend running a few real-world extracts from within the VieleRETS interface, this time checking that “Update Batch Job” box in the extract window. Once the job has completed, navigate to the /vieleRETS/batch_control_files folder via FTP, and grab a copy of the updated batch file so you can get a clear understanding of what naming conventions your RETS server is using for things like price and Status, not to mention how they are spelling the cities in your region.
Armed with this information you can begin carving out your own batch files. Just make sure you save all your batch files to the /vieleRETS/batch_control_files directory for use with CRON
The CRON Script:
Now that you’ve compiled your list of batch files, now you need to create your cron script.
It’s a plain unix file with an .sh file extension.
I called mine rets_cron.sh
Here’s how it looks:
#!/bin/sh
cd /home/your-hosting-account/public_html/vieleRETS-1.1
#residential
php -q ./run_interactive_job.php batch_control_files/residential _1
echo Updated residential_high_1
php -q ./run_interactive_job.php batch_control_files/residential_ 2
echo Updated residential_high_2
php -q ./run_interactive_job.php batch_control_files/residential_ 3
echo Updated residential_high_3
The jist here is that you’re going to list every one of your batch files here, in this format. The “echo” line tells the script to write “updated x” to a log file which gets emailed to me every time the batch runs, you can set this up in cron.
Once you’ve added all your batch files to the script, save it to your web server, you can stick it anywhere you like, but I like to keep it at the root of the directory structure…it makes for less typing when I go to make my cron job.
The CRON JOB
I really wouldn’t recommend running the job more than once a day, even that could be a little excessive, depending on the size of your market. You run the risk of really milking your bandwidth, and if you are in a big market, its going to take a LONG time to download everything, possible degrading server performance during peak hours.
I set mine to run at 3am every weeknight. So the Cron schedule looks like
* 3 * * *
The command run by cron will depend on where you placed your cron script, mine looks like:
/bin/sh /home/your-hosting-account/rets_cron.sh
(note the space between the /bin/sh and the /home)
One thing I’ve done is adjust the time of the cron job to fire off 10 minute from now, that way I don’t have to wait till tomorrow to find out the hard way that I’ve blown something. You can always adjust the schedule of the job again later.
I’ll be trying to update/edit this from time to time. I’ll try to answer questions if they come up, I’m not a great teacher, but I hope this helps someone hit the ground running. I also posted this same post to my blog, here (http://www.tarpontech.net/vielerets-automatically-updating-wp-realty/), that has a few screenshots from within VieleRETS in case that helps folks get their bearings a little better. Good luck!
What this covers:
1. How to configure VieleRETS for use with WP-Realty
2. How to configure a batch job in VieleRETS
3. How to configure a batch job to run in CRON
What this does not cover:
1. How to install VieleRETS and perform general configuration (i.e. communicating with your RETS server)
2. How to install or configure WP-Realty
For general VieleRETS setup information,Trent Richardson (http://trentrichardson.com/2008/02/06/rets-retrieval-with-php-and-vielerets-part-1/) created a terrific blog postthat I found invaluable when starting out.
Creating Sources: Telling VieleRETS what to download, and planning ahead for WP-Realty integration.
Setting up your initial sources can be…problematic if things don’t go well out of the box, but between Trent Richardson’s blog posts and the VieleRETS message board, hopefully you’ve been able to get communication up and running between your RETS server and VieleRETS. We’re going to focus on issues that directly pertain to getting VieleRETS and WP-Realty to play nice.
The first thing to know is that you will need to create a Source for each of the property types you wish to display listings for, examples being Residential, Commerical, and Lots (land). Make sure to name your Sources per Property type, don’t just call them Default, Default_1. Name your Sources Commercial, Residential…ect…you’ll thank me later.
You’ll notice that part of the process of creating a source is choosing which fields to download. It’s entirely up to you what fields you wish to use and/or display, but take note of what you pick. You’ll need to create the listing fields within WP-Realty that correspond to the fields you are downloading from VieleRETS.
This step is VERY, VERY important. Take your time, think it out, consult with your clients/agents/friends/family, whatever, but don’t take this step lightly. Save yourself A LOT of additional time and work, and make certain you’ve selected every RETS listing field you may, possibly, ever need, and make certain that you’ve created a corresponding listing field inside WP-Realty.
Missing a field at this stage will cause you to have to start all over again if you decide you really needed that “other rooms” field a month from now. You’ll need to purge your database, recreate all of your VieleRETS configuration, and then download the whole database again.
The fields inside WP-Realty don’t need to match exactly what they’re called in RETS, such as, the field in RETS is called Subdivision, I called it Neighborhood in WP-Realty, but it’s recommended to keep things the same. It makes things a lot easier when you are sorting through 120 fields to match them up. You’ll see why when we get to creating Extracts.
Lastly, while I can’t help with the myriad different ways VieleRETS talks to different RETS servers, I will give a little advice. RETS servers like for folks that are querying it to ask for certain minimum information. In my case, most servers like you to at least query for City, ListPrice, and Status. If you can’t seem to download anything, I’d check to see what fields you are using to Query, first.
Ok, so, we’ve got Sources set up to match all the Property Types we’ll be using, and we’ve selected all the fields we’re going to download for each Source, and we’ve created all the corresponding fields inside WP-Realty.
Creating a TARGET: Telling VieleRETS how to find WP-Realty.
VieleRETS needs the full root path to your wp-realty plugin installation. How to determine this depends on your hosting environment. Most shared web hosting accounts stuff all of your files inside of your account’s /home/ directory, but if you are unsure of your root path, save yourself much time and aggravation and call your hosting tech support and get the right answer.
Once you’ve determined the root path to your wp-realty plugin installation, you have two possible ways of using VieleRETS’ menu to navigate to it.
If the server’s permission structure allows, you can just click the radio buttons corresponding to the correct folder you need to navigate into, or, more reliably, just type the full root path into the field labeled
“Full path to Open-Realty install”.
It should look something like:
/home/your_hosting_account/public_html/wp-content/plugins/wp-realty
One little trick about VieleRETS is that it may not automatically recognize the path once you’ve typed it in.
By clicking someplace else on the page, such as clicking into the Name field at the top (just don’t change anything!), VieleRETS usually refreshes and will navigate to the path you’ve specified. When this happens, the screen will change to display the list of WP-Realty fields available to be populated by MLS data.
If you don’t see the screen change to show these fields, you most likely have a file permission or php permissions issue, both of which are outside the scope of this document.
Let’s hope you didn’t have a permissions issue. I’ve used this system on 4 different hosting accounts, with different results, but when permissions and PHP configuration are correct, creating a Target is dead easy.
Creating an Extract: Getting VieleRETS and WP-Realty to play nice.
Once you’ve created your individual Sources within VieleRETS, you’ll need to set up your Property Types within WP-Realty, using the EXACT names being used by VieleRETS. Doing this while creating your Extracts is a good idea, as you’ll have the Property Type names (called Class Name Style in VieleRETS) staring you in the face during the Extract creation process.
Word of advice: If VieleRETS is using Residential, don’t create Property Type residential in WP-Realty, it’s case sensitive.
Once you’ve established what property types you’re going to use, take note of the Property Type names in use by your MLS. My best example of this is in regards to Multi-Family units such as condos. Many RETS servers will provide you with the option to use Condos/Townhomes or CommonInterest as possible property type names. The WP-Realty database does not play well with Property Type names that use a slash, such as Condos/Townhomes, so you’ll want to choose CommonInterest, or another Property Type name that doesn’t use a slash. Remember, Case Sensitive.
Creating an Extract isn’t difficult, but it is tedious, which is why its so important to have all your fields set up correctly in the Source, and in WP-Realty, not to mention having everything named right.
To start, name your extract per Property type, as well, you’ll need one extract per property type. Each Extract reads from its own Source, but luckily they all point to the same, shared Target (wp-realty’s database).
You’ll need to choose the corresponding Source, (see why naming is important?), I’d take a moment to consider the “maximum images to download” option. Something around 10 images is probably more than enough to displaying a property. Some overzealous realtors can stuff a lot of images on the MLS for a single home, and your server’s storage capacity and bandwidth limitations won’t be so kind to you.
The last setting on the main Create an Extract page is the Class Name Style. We’ve already gone over this at the beginning, but be sure to pick a class name (property type name) that plays well with the WP-Realty database.
Once done with this page you can apply your changes, and you’ll be confronted with a menu of options.
Now is the time for some tedium. The only other area we need to focus on for an Extract is the Data Map. Don’t mess with the (Expert) data map. The Data Map page is self explanatory. Here you tell Viele-Rets where inside the WP-Realty database to put the data it is downloading. If you followed my advice and named your WP-Realty fields the same as the RETS fields, it’s a simple matter of matching names. Once done, triple check, however. You don’t want prices to start showing up where half-baths are supposed to display.
That’s really it for creating an extract. Now you just have to do it a few more times for each Property Type. Enjoy.
The moment of truth
Now we find out if we have any idea of what the hell we’re doing. Time to run an extract.
From the VieleRETS main menu, choose Download Listings.
Choose an extract and Apply.
Your download Criteria are dictated by the “Fields to Query” you set up during Source Creation.
If you don’t get any results, you may have to fiddle with these fields till you get something your RETS server likes.
In this test case, I’m using Status, City, and ListPrice. You may want to “Set Download Limit” to 10 or 15 listings to start, we’re just trying to make sure the system works, we can pull the entire contents of the MLS after we know things are a go.
The last thing I want you to take note of before we apply our settings to run an extract is the “Update the Batch Control File” Later on we’ll be talking about automated batch jobs, and using this feature can save us a little time in creating our batches. For now, leave the box checked.
If things went well, the batch will complete, and you’ll see a brief summary displayed at the bottom of the page detailing how many listings and images were downloaded ect…
If you didn’t get a successful extract, but you’ve gotten this far, you probably (hopefully) just need to tweak your “Fields to Query” in your Sources, and adjust what criteria you use when running the extract.
For those of you that got a successful batch, you should be able to view the listings inside your Wordpress blog, depending on how you’ve set up WP-Realty. Or, you can view the listings under the “Edit My listings” or “Edit All Listings” inside the WP-Realty Control Panel.
If you’ve got properties, you’re in business.
Creating Viele-RETS batch jobs and running them automatically via CRON
I’m skipping ahead a little bit, and I’m not going to go over how to run a batch from the command line, the VieleRETS message board and Documentation goes over that well enough. What we’re going to talk about is how to fill up your wordpress blog with every single listing your RETS server will allow you to download.
It’s actually really easy, just takes some busy work. First, lets look at a sample batch job:
<VIELE_BATCH>
<EXTRACT> residential
</EXTRACT>
<!--<QUIET/>-->
<UPDATE_BCF/>
<!--<UPDATE_ONLY/>-->
<LIMIT> NONE</LIMIT>
<QUERY_VALUE name="ListPrice">400000+
</QUERY_VALUE>
<QUERY_VALUE name="ListingStatus">|A
</QUERY_VALUE>
<QUERY_VALUE name="ListingID">
</QUERY_VALUE>
<QUERY_VALUE name="ModificationTimestamp">
</QUERY_VALUE>
<QUERY_VALUE name="City">|Watoosee
</QUERY_VALUE>
<LAST_RUN>2010-04-20T17:28:33</LAST_RUN>
</VIELE_BATCH>
If you stare at it long enough, you’ll see that by running this batch job I’ll be running an extract on the Residential property type for Active listings over $400,000 in the city of Watoosee.
The basic premise is that you’ll have to create a batch job file for every query you wish to run.
In my particular RETS, I have 3 price categories, -$300,000, $300,000-$400,000, and $400,000+.
If I want to download all active Residential listings in Watoosee, I have to create 3 batch jobs, one for each price range. Want Commercial Properties in Watoosee? That’s three more files. Yes, you’ll be making lots of batch files. Not only making them, but please name the files something intelligent, as well. You’ll need them named appropriately for adding to your cron script, and for logging and troubleshooting.
You can get started by taking this batch file, however, I recommend running a few real-world extracts from within the VieleRETS interface, this time checking that “Update Batch Job” box in the extract window. Once the job has completed, navigate to the /vieleRETS/batch_control_files folder via FTP, and grab a copy of the updated batch file so you can get a clear understanding of what naming conventions your RETS server is using for things like price and Status, not to mention how they are spelling the cities in your region.
Armed with this information you can begin carving out your own batch files. Just make sure you save all your batch files to the /vieleRETS/batch_control_files directory for use with CRON
The CRON Script:
Now that you’ve compiled your list of batch files, now you need to create your cron script.
It’s a plain unix file with an .sh file extension.
I called mine rets_cron.sh
Here’s how it looks:
#!/bin/sh
cd /home/your-hosting-account/public_html/vieleRETS-1.1
#residential
php -q ./run_interactive_job.php batch_control_files/residential _1
echo Updated residential_high_1
php -q ./run_interactive_job.php batch_control_files/residential_ 2
echo Updated residential_high_2
php -q ./run_interactive_job.php batch_control_files/residential_ 3
echo Updated residential_high_3
The jist here is that you’re going to list every one of your batch files here, in this format. The “echo” line tells the script to write “updated x” to a log file which gets emailed to me every time the batch runs, you can set this up in cron.
Once you’ve added all your batch files to the script, save it to your web server, you can stick it anywhere you like, but I like to keep it at the root of the directory structure…it makes for less typing when I go to make my cron job.
The CRON JOB
I really wouldn’t recommend running the job more than once a day, even that could be a little excessive, depending on the size of your market. You run the risk of really milking your bandwidth, and if you are in a big market, its going to take a LONG time to download everything, possible degrading server performance during peak hours.
I set mine to run at 3am every weeknight. So the Cron schedule looks like
* 3 * * *
The command run by cron will depend on where you placed your cron script, mine looks like:
/bin/sh /home/your-hosting-account/rets_cron.sh
(note the space between the /bin/sh and the /home)
One thing I’ve done is adjust the time of the cron job to fire off 10 minute from now, that way I don’t have to wait till tomorrow to find out the hard way that I’ve blown something. You can always adjust the schedule of the job again later.
I’ll be trying to update/edit this from time to time. I’ll try to answer questions if they come up, I’m not a great teacher, but I hope this helps someone hit the ground running. I also posted this same post to my blog, here (http://www.tarpontech.net/vielerets-automatically-updating-wp-realty/), that has a few screenshots from within VieleRETS in case that helps folks get their bearings a little better. Good luck!