TensorFlow Time Series Tutorial Enhancement Gone Wrong

R

raeldor

Guest
raeldor Asks: TensorFlow Time Series Tutorial Enhancement Gone Wrong
I’ve been following this time series tutorial for Tensorflow…

https://www.tensorflow.org/tutorials/structured_data/time_series

And it was going well and seemed to work ok. I substituted with my own dataset (about 1.1m records with 4 features) and it also seemed to work well, but the memory was getting REALLY tight, so I thought I would try and implement the improvement mentioned at the bottom that said…

In addition, you may also write a generator to yield data (instead of the uni/multivariate_data function), which would be more memory efficient. You may also check out this time series windowing guide and use it in this tutorial.

This kind of made sense, as it seems to produce a sliding time window with single steps so the data becomes HUGE. After reading through this…

https://www.tensorflow.org/guide/data#time_series_windowing

It looked like I could use tensor flow to do the windowing, which I assumed would get executed dynamically therefore not much memory would be used. After reading, I came up with the following code to try and replace the multivariate_data function (functions are borrowed from the tutorial, though I changed the dense_1_step to get a single set of the label features back instead of a window…

Code:
def make_window_dataset(ds, window_size=5, shift=1, stride=1):
  windows = ds.window(window_size, shift=shift, stride=stride)

  def sub_to_batch(sub):
    return sub.batch(window_size, drop_remainder=True)

  windows = windows.flat_map(sub_to_batch)
  return windows

def dense_1_step(batch):
  # Shift features and labels one step relative to each other.
  return batch[:-1], batch[-1:]

# get training samples (features and labels)
train_ds = make_window_dataset(tf.data.Dataset.from_tensor_slices(dataset[:TRAIN_SPLIT]), window_size=past_history+1, shift = 1, stride=1)
dense_labels_train_ds = train_ds.map(dense_1_step)

# get validation samples (features and labels)
val_ds = make_window_dataset(tf.data.Dataset.from_tensor_slices(dataset[TRAIN_SPLIT:]), window_size=past_history+1, shift = 1, stride=1)
dense_labels_val_ds = val_ds.map(dense_1_step)

# batch and shuffle training data
train_data_single = dense_labels_train_ds.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()
val_data_single = dense_labels_val_ds.batch(BATCH_SIZE).repeat()

But for some reason, it's all gone pear-shaped for the following reasons...

  1. The training loss and validation loss are now NOTHING alike and in fact, I don’t think the net is even learning properly anymore
  2. The memory usage seems much better UNTIL you execute the fit() and then it just blows up about the same anyway.
  3. I get the warning ‘Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.’

I’m sure I screwed something up here, but to me, it looks like it's doing what it should be doing. I'm still on a steep learning curve with this, but if anyone with more experience could point me in the right direction I would be very grateful.

Thanks.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.
 

Unreplied Threads

How to perform feature engineering on columns with multiple categories?

Prateek Coder Asks: How to perform feature engineering on columns with multiple categories?
I want to perform feature engineering on a data that mostly contains textual data and lot of columns with multiple categories like Supervisor, location code, employee class, business unit, job title, etc. What should be the best approach to do this? I have binary encoded the column with 2-3 categories or performed one hot encoding, but these columns contains different categories from 8 to 150. Different location will play major role in prediction, so will the employee class or business unit etc and I believe they will be very important for my model.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

[Solved] Create Buffer but keep original Attribute table

J.Beer Asks: Create Buffer but keep original Attribute table
I want to create a buffer around a number of features. It is important for me to keep the attributes of the features in the new buffer.shp layer. When I try to write a new VectorLayer with a QgsVectorFileWriter, I do get the buffered polygons and also the fields of the original layer get copied over, however the attribute table stays empty.

How can I copy over the attributes of the original layer to the buffer layer?

As you can see I already tried to load the attributes using the . attributes() method on the QgsFeature object and setting the attributes. However I feel I am missing something here.

Code:
for f in feats:
    attributes = f.attributes()
    f.setAttributes(attributes)

See full code below:

Code:
path = '/Volumes/VAW_ETH-Z/M.Thesis/data/geospatial/'

# load filtered sgi
sgi_filtered = QgsVectorLayer(path + 'sgi_filtered.shp')

## Create buffer around filtered sgi
outFn = path + 'gl_buffer3.shp'
bufferDist = 50 # in map units

fields = sgi_filtered.fields()
feats = sgi_filtered.getFeatures()

writer = QgsVectorFileWriter(
            outFn,
            'utf-8',
            fields,
            QgsWkbTypes.Polygon,
            sgi_filtered.sourceCrs(), # loads the reference system of the source feature
            'ESRI Shapefile')

for f in feats:
    attributes = f.attributes()
    f.setAttributes(attributes)
    geom = f.geometry()
    buff = geom.buffer(bufferDist, 5)
    f.setGeometry(buff)
    
    writer.addFeature(f)

Screenshot of the Attribute Table below (it shows the fields but no attributes):

Empty Attribute table. The fields are displayed however no attributes have been added.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your response here to help other visitors like you. Thank you, solveforum.

[Solved] Saving a Project Folder as a template (QGIS 3.28)

  • Ben Sear
  • Geography
  • Replies: 0
Ben Sear Asks: Saving a Project Folder as a template (QGIS 3.28)
I am looking to create a template for easy layout integration for new QGIS project files, but when saving the project file to a template using save to > template, the file does not appear in the project templates as a template file, but just a new project file. I read that you need to restart GIS once you saved a project file as a template but restarting does not change the project file to being a template. Has anyone experienced saving template before and have any suggestions? Thank you.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your response here to help other visitors like you. Thank you, solveforum.

Method of cooling a gas without liquid evaporation for Sci-fi novel

  • user2765977
  • Chemistry
  • Replies: 0
user2765977 Asks: Method of cooling a gas without liquid evaporation for Sci-fi novel
sorry for the amateur question here.

I know little about science, but regardless am writing a fictional amateur novel that deals with a lot of chemistry stuff, especially gasses.

My protagonist needs a clever method of cooling a gas. He has access to several gasses (CO2, O2, H2, nitrogen) in varying temperatures (70c and up). Unfortunately no water (at least not more than a few bottles). The atmosphere he is working under (planet atmosphere) is in the range of 150c and at a pressure of roughly 150kpa in the novel.

He also has access to sci-fi style manufacturing machines that can simply 'make' parts, like pipes, valves, pumps, that sort of thing.

I am looking for a clever way or really any way at all, that he can use to cool down some O2 to at least under 40c in such a situation.

My amateur reading online suggests the main way, is via liquid evaporation, but I doubt that would work in the above scenario easily.

Does anybody have any ideas? Is this even possible?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

Is there a "fundamental problem of thermodynamics"?

  • agaminon
  • Physics
  • Replies: 0
agaminon Asks: Is there a "fundamental problem of thermodynamics"?
The "fundamental problem of mechanics" can be boiled down to finding and solving the equation of motion of a system. Similar statements can be said for quantum mechanics for the Schrödinger equation and for electrodynamics and Maxwell's equations, etc. But is there such a thing for thermodynamics? Is there a formulation that allows for this kind of perspective?

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

What does this vertical line notation mean?

  • Lambda
  • Physics
  • Replies: 0
Lambda Asks: What does this vertical line notation mean?
Here is the definition of the Noether momentum in my script.

$$I = \left.\frac{\partial L}{\partial \dot{x}} \frac{d x}{d \alpha} \right|_{\alpha=0} = \frac{\partial L}{\partial \dot{x}} = m \dot{x} = p_x.$$ But I don't understand exactly what this vertical line with $\alpha=0$ means, I would have interpreted it as $\frac{dx}{d0}$ now.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.

Does diffusion of a spring depend on the spring’s equilibrium length?

  • Robert S. Pierre
  • Physics
  • Replies: 0
Robert S. Pierre Asks: Does diffusion of a spring depend on the spring’s equilibrium length?
Say we perform a random walk on $\mathbb{Z}$ with two equal particles that are connected by a spring with spring constant $K$ and equilibrium length $a$, where at each time step each particle has probability $\frac{1}{2}$ of making a step $\pm1$. However, at each step a random number $q\in[0,1]$ is sampled in order to refuse the move if $$q\geq\exp\left(\Delta E\right),$$ where $\Delta E$ is the difference in elastic energy between the two time steps.

Does the diffusion coefficient depend on the equilibrium length $a$?

I know that the diffusion is independent of the spring constant $K$, so we find the same diffusion coefficient for all $K$, namely that of two independent particles. I assume the same holds for the equilibrium length, but is this true and how can I see this?
And furthermore, what role does the initial extension of the spring play? Say the initial length of the spring $x_0$ is not equal to $a$, then the whole system will need a while to equilibrize, as opposed to when we start with $x_0=a$.

SolveForum.com may not be responsible for the answers or solutions given to any question asked by the users. All Answers or responses are user generated answers and we do not have proof of its validity or correctness. Please vote for the answer that helped you in order to help others find out which is the most helpful answer. Questions labeled as solved may be solved or may not be solved depending on the type of question and the date posted for some posts may be scheduled to be deleted periodically. Do not hesitate to share your thoughts here to help others.