ESdat Forums
»
Forums
»
ESdat
»
Uploading Historical Data
Rank: Member
Groups: Registered
Posts: 12 Location: Calgary, Canada
|
Hi, So I am having some issues uploading historical data - I've used the 'chem column format' template before with this project but for some reason this time it's not working. When I hit upload I get the expected 'samples' and 'results' tables. Samples uploads fine but when I move to the results I get some confusing errors.... First I will set the import options to 'Add new' and 'Create Missing Parent Records' this returns errors for all records that the records already exist. It gives the same error if I use 'Require Parent Records'. If I choose 'Update Existing' / 'Require Parent Records' the error I get is the records don't exist. I get the same error if I choose 'overwrite existing'. How is it that my records both exist and don't exist at the same time! quite the paradox.... ugh. I've tried unhighlighting all coloring and removing all error comments; I've tried recreating my entire spreadsheet. Any hints/suggestions??
|
|
|
|
Rank: Administration
Groups: Registered, Administrators Posts: 259 Location: Ballina
Was thanked: 8 time(s) in 8 post(s)
|
Hello, Would you be able to let us have a look at the historical data Import Template? Many thanks, Alex
|
|
|
|
Rank: Member
Groups: Registered
Posts: 12 Location: Calgary, Canada
|
Here's one portion of my data. I start with the 'Chem Column' sheet and then click on the import button (little blue arrow) and it produces the 'Samples' and 'Results' sheets.... as you can see from the color the samples sheet imports and then the results sheet first gives the 'already exists' message and then changes to 'doesn't exist' (as described in post above). File Attachment(s): historical_data.xls (1,333kb) downloaded 9 time(s).You cannot view/download attachments. Try to login or register.
|
|
|
|
Rank: Member
Groups: Registered
Posts: 12 Location: Calgary, Canada
|
Just wondering if anything had come of the investigation of my data template? it's crunch time for getting this data into ESdat; open to suggestions on a workable solution.
|
|
|
|
Rank: Administration
Groups: Registered, Administrators Posts: 259 Location: Ballina
Was thanked: 8 time(s) in 8 post(s)
|
Hello, Im having trouble opening it at the moment. One thing to check though, have you populated all the columns with a double green column heading (the combination of w=which define a unique record - the primary key). In particular the "Total_or_Filtered" and "Method_Name" columns in the Chemistry Results Sheet. Alex
|
|
|
|
Rank: Administration
Groups: Registered, Administrators Posts: 259 Location: Ballina
Was thanked: 8 time(s) in 8 post(s)
|
I managed to open the spreadsheet, and my previous recommendation applies (also check the Result Type is populated). When doing updates of chemistry results it is possible the system will find two occurances of an equivalent record if all the primary key values aren't specified, as the update searches for an existing record which satisfies that combination of primary key values. This is apparently what has happened here. It looks like you may be trying to completely overwrite the entire dataset previously loaded. If so, another, better approach, is to delete the data that was previously imported into the database and re-import. This is particularly pertinant if you have edited any of the primary key values. This can be made easy by specifying a "DataSource" value for the Samples, you can then filter for those samples using the DataSource value, delete them (which will also delete associated results) and re-import. Alex
|
|
|
|
Rank: Member
Groups: Registered
Posts: 12 Location: Calgary, Canada
|
Thanks for getting back to me; Gives me things to watch for going forward. With the delay in response I actually did go back and deleted all added to this project and then re-uploaded. It seems to work better now. I will go back and make sure all the double green fields are filled in. I do have a couple of annoyances to note though - when you first get this historical chem table from ESdat some of the yellow fields have the option to change the field header through a drop down menu (ie: SDG can be changed to Lab_Report_Number) however when I do this and then upload, the data from the field that was changed doesn't upload - but if I leave it as is (ie SDG) then it will upload and I then move this information to the correct field in the samples table. Another buggy thing is when I'm entering sample type information on the samples table I get a drop down for field_D, field_B etc... it asks me to select the parent sample and puts that information in the next field (parent sample). However when I have literally thousands of records this is not very helpful - i skim through to find the next dup and when I find that box there is no drop down - i can type in field_D but the selector doesn't come up until I have tabbed or press enter - then the selector drops the parent sample info in the the box beside (ie not in the parent sample field or in the record below the record of interest) - it would be preferrable if the selector didn't exist or is activated without leaving the cell where I'm entering field_D. Obviously I've found ways to work around both of these but they are irritating when you have so much data to deal with.
|
|
|
|
Rank: Administration
Groups: Registered, Administrators Posts: 498 Location: Byron Bay Was thanked: 19 time(s) in 19 post(s)
|
Hello,
With regards to the drop-downs that are available in the Excel Import Templates, by default only the first 100 rows have them. If you want the drop-downs for rows after that you could copy and paste the top rows over as big a range as you need.
The thinking in not populating more than 100 with drop-downs is firstly avoiding generally unnecessary bloat in the Excel file. Also though, historical data of that volume is more efficiently maniupulated from the original source using some cleansing mechanism (manual or automated) and then dropped into the import template as a complete dataset. If that isn't suitable, then as above you can copy rows down to get as many rows with drop-downs as you need. I used your Import Template previously attached to try to replicate the issue when changing the column headers. It didn't have a SDG Column so I changed Sample_Type to Monitoring_Round. When I imported and the data was copied from the "Chem Column Format" sheet to the "Chemistry Samples" sheet as a preliminary to importing the data was copied correctly into the Monitoring_Round Column. I may have misunderstood where the issue was, so if that is the case, can you let me know. Thanks, Warwick
|
|
|
|
ESdat Forums
»
Forums
»
ESdat
»
Uploading Historical Data
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.