Hi Howard,
yes the not using the spatial index is some of the things I discussed in my post gis.thinkgeo.com/Support/DiscussionForums/tabid/143/aff/12/aft/7864/afv/topic/Default.aspx before.
In my code I’m using the spatial Index, with the query I posted in the forum.
I think there is something wrong when you are rendering the things, as I post before using shapefiles is also very slow comparing OpenSource Desktop QuantumGIS and yours WPF API.
Could it be that you are loading to much information to memory? Are you loading also alphanumeric data when rendering?
Memory and CPU problem
Hi Rui,
So the research is still in progress. As far as I know, we only query the features in side of current viewport extent; then save it into the memory. Here are two things might cause the high CPU and memory usage. One is the viewport extent is pretty big so that, all features on your layer are be queried. While another reason is that the querying doesn’t work with the index which makes low performance. Thanks to your advice.
I guess, slow for shapefile is because the projection you added. Would you mind to remove the projection and set simple style instead to test. On the other hand, did you try WPF edition or WPF Desktop Edition? WPF Desktop has better user experience, please have a try.
Thanks,
Howard
when you say: "then save it into the memory.", are you fetching into memory all columns on the table? Or only the Geometry?
Rui,
That’s a good question. When we request features from a layer, the column values are requested when we need get them. For example, when you call get features method, we usually have two parameters. One is viewport extent to request features, another one is the retrieve column value condition; such as all columns or some specified column names.
During rendering, we don’t get all columns back if the column value is not needed; in another word, most of time, only geometry and feature id is requested.
Thanks,
Howard
Well, those are not very good news :(
And the use of COM objects? You are using some, are they being free from memory?
Hi Rui,
Currently, all for OracleFeatureSource is managed code. Just as Howard has mentioned, the low performance is caused by the spatial index, and we can address that the increased memory comes from the conversion between different projections, but we are still on the road.
Thanks,
Johnny
But is the projection related code also in managed code?
Rui,
Definitely, Yes. But we are getting closer to the end. And any result will be posted here.
Thanks,
Johnny
Any progress?
Hi, Rui
Here’s the latest news about the memory and CPU consuming problem. We’ve tested the following classes in MapSuiteCore for Projection. ManagedProj4Projection, UnmanagedProj4Projection, Proj4Projection. We found that the memory consuming problem occurred in all of them.
However, while using ManagedProj4Projection or UnmanagedProj4Projection, the conversion is much faster than using Proj4Projection.
Here are two sets of data coming out of our tests with the shapefile you provided in your sample.
Test1:
Use code sample you provided: ThinkGEOWEB_MemoryTest( 4 browsers to simulate 4 requests)
ClassName
Time First Imaged Returned(sec)
Memory(MB)
CPU(%)
Proj4Projection
45~60
200 ~700
25~70
ManagedProj4Projection
<30
200~700
25~70
UnmanagedProj4Projection
<30
200~700
25~70
Test2:
Just call GetAllFeature method in MapSuiteCore.
ClassName
Time All Feature Returned (sec)
Memory(MB)
Memory After Calling GC.Collect (MB)
CPU(%)
Proj4Projection
26
180
14
25
ManagedProj4Projection
14
180
14
25
UnmanagedProj4Projection
14
180
14
25
So, you can have a try on both ManagedProj4Projection and UnmanagedProj4Projection to see if it helps to improve the performance.
The other issue you’ve mentioned in the first post: “memory overflow, ORA: 1426”. Is it possible that the error is caused by some large geometry data?
We had encountered this kind of error when we were preparing some large data tables for Oracle Spatial. For example, like a record of a MultiplePolygon has more than 8,000 points in it, when calling Get_WKB function on this field, the same error is raised at the server side, in SQLPLUS.
We’ll keep on working to solve the memory consuming problem. That’s for your information and advice.
Thanks.
Hello,
I have a column with the WKB pre-generated since sometimes the oracle's get_wkb() just takes to long. So when I get the WKB I'm really getting it from this pre-generated WKB. Then I have some PL/SQL that updates the WKB value as needed (for example: insert or updating).
The memory overflow is not caused by large geometry data. When it happens, I just refresh the layer and it shows OK. And only happens after a while working on the map. Is like it's related with the memory usage.
Since both services are in the same server (Oracle & IIS) could be this? ThinkGeo just sucks all the memory and there is some need on Oracle's side, could it be?
I've tested with a separated server, and this error don't appears, but thinkgeo started complaining about the WKB not being well formatted, although when I refresh the layer everything is nice again.
I really think that all this problems are related with the amount of memory consumption by thinkgeo.
Hi, Rui
Thanks for providing the information. I took your advice and tried to enlarge the memory of Oracle 10g server (In the past it was 1G, and now it’s 2G). It did help to reduce the frequency the error I mentioned in the earlier post, somehow, I still can’t get rid of it. At first I thought it probably was caused by the memory pressure during the querying, but after checking the performance of the Database Server, I found total memory usage just stayed at about 900MB. I can’t help but wondering if this kind of data is suitable for Spatial Querying, additionally, it’s weird that it seems this kind of error occurs randomly, it fits for the description in your post (At first, it works fine, after zooming in and out, the error is raised) .Here is the error message I got, I don’t know if it’s the same with the one you got in your application. It’s probably thrown at the server side and caught by the System.Data.OracleClient.
ORA-29532: Java call terminated by uncaught Java exception: java.lang.OutOfMemoryError
ORA-06512: at "MDSYS.SDO_UTIL", line 1960
ORA-06512: at "MDSYS.SDO_UTIL", line 1982
ORA-06512: at "MDSYS.SDO_GEOMETRY", line 30
Here’s another thing about the memory problem. We’ve found out the it’s the projection and the way we handle projection caused the large memory consumption, however, it’s not a memory leak, since when the memory pressure goes up, GC will recycle the garbage on the LOH, where most of the WKB bytes go (For the shapefile you provided in your sample, about up to 100MB), manually call GC.Collect will also have the memory pressure reduced.
We’ve created an issue about the memory consuming problem in our own issue system, and we’ll let you know if there’s any progress in fixing this problem.
I beg your pardon for keeping you waiting for so long, and I really appreciate if you would provide us further information.
Thanks.
James
Hello James,
Yes one of the error types I'm getting is that one, but is a little random the error type I get.
I'm a bit worry about this, since I have the project to deliver and I can't deliver it having this major problem. This will bring some problems to my company. I really don't know what to do... I wish I could help you solve this problem, but I really don't know what I can do.
I don't know what you guys are doing inside your code.
Will it help if I post a ticket?
Rui,
Don’t worry about this issue, I think we can figure out a good solution for it together.
As we know, the problem is mainly caused by using projection on the fly, so my suggestion is that could you project the data first and insert to Oracle database, so all the feature are already projected, when we get them from database we don’t need to project again. We provide the API for editing, you can find the sample in our HowDoI sample “Editing Feature Layers”, it shows how to add/update/delete feature to InMemoryFeatureLayer, the OracleSpatialFeatureLayer will follow the same way.
You don’t need to post a ticket, we will keep focusing on this thread until your problem solved.
Thanks,
James
Hello James, I already suggested that to our customer, but he doesn’t agree. The reason is simple, we have several maps with different projection. For instance, EPSG:900913, EPSG:3763, … and like this we would have a lot of repeated data in the oracle DB. In a small DB this wouldn’t be a problem, but in this case we are talking about a database with 1GB more less.
So please, try giving some priority to this as you can… Thanks
Rui,
We will think another way to make it. Do you think if we provide our source code of OracleSpatialFeatureLayer and OracleSpatialFeatureSource will give your some help? If it’s so, please contact support@thinkgeo.com to request it.
Thanks,
James
Hello James, you already did that, and I already made some optimizations on loading the data, but the memory usage problem come from parts of thinkgeo core that we don’t have.
Rui,
Everything comes to him who waits. Eventually, we found a solution to optimize memory usage for projection operation. The test result is obvious, before the changes, the peak memory cost 160MB with your test data, and we create 20 threads to consume your data at the same time, the OutOfMemoryException was always thrown; after the changes, the peak memory usage is less than 100MB, and we can run application properly when we create 20 threads.
A little pity thing is that you couldn’t get the daily build package immediately because we are upgrade our building utility, at soonest you will get it at the day after tomorrow which version is 4.5.107.0
Thanks,
James
Seems to be better now… But we are now running some quality tests. Since we had problems with a customer of ours related to the use of Thinkgeo products, we now are more carefull with the publication of ThinkGeo products versions in production enviroments.
For what comes with this concrete problem it seems to be a bit better now. But has we run intensive tests we are finding more problems. Now we have this same problem but with the use of Geo/TIFF raster data.
We now are having a lot of problem with this. We have ± 1TB of aereal imagery that we just can’t load it into the application. Even with less than 1GB of imagery the aplication is crashing after 2 or 3 zooms.
So at the end, there is still some memory consumption related problem in thinkgeo and now is not related with Projection. But we are preparing some example to send to you guys.
Thank you for everything.
Rui,
Let's focus on the post 8829 now and hope the new assembly will be in production soon.
There is a known memory issue about GeoTiff and we will work on that, here is the related post, have a look if intrested.
gis.thinkgeo.com/Support/Dis...aspx#23756
Thanks,
Ben