Trigger
#
Every daily at the end of the (work) day.
Duration
#
15 minutes.
Steps
#
- Review what was planned for the day (manual)
- Did you finish each planned task?
- Were any deliverables missed or delayed?
- How many times I was interrupted?
- How much time I spent on unexpected work?
- Was I got blocked and for how long?
- Rate the quality of work delivered.
- Provide feedback related to the plan (manual)
- Learning & Insights (manual)
- Did you discover a new tool, technique, or process?
- What could be improved next time?
- Review weekly plan and align
- Plan next day
Trigger
#
Every week, either at the beginning or end of the week.
Duration
#
15 minutes.
Steps
#
Overview
#
In this article I list metrics and alerts one should have when monitoring a GPU cluster to ensure efficient utilization of resources.
GPU cluster monitoring is critical for organizations to optimally utilize the limited capacity they have.
Without monitoring it is easy for users to leave jobs running that do not use GPU resources, or do not use them efficiently.
In some cases GPU clusters use certain technologies that require the users to provide images with specific libraries, and not including those dependencies can result in significantly worse compute performance.
Metrics
#
- Allocated GPUs
- Used to determine who (or which project) has GPU allocated (i.e., currently assigned to a running workload)
- GPU utilization
- Used to determine whether the GPU is partially or fully used, and if it is partially used, to potentially identify the causes
- GPU memory utilization
- Used to determine if the GPU memory is partially or fully used
- Used to identify out of memory issues and potential memory leaks
- InfiniBand receive/transmit bytes
- Used to determine if a workload is making use of the technology
- Job launch wait duration
- Used to determine when there’s queueing of jobs due to compute being exhausted and how long it takes for jobs to start
- Job duration
- Used to gather statistics about the type of workload running on the cluster in order to make informed decisions
Alerts
#
- Allocated GPUs are used
- Used to detect jobs that may ask multiple GPUs but end up using 1 or only a few of them
- GPU utilization below threshold (<10%)
- Used to detect workloads that do not make full use of the GPU or are allocated to an oversized GPU
- GPU utilization above threshold (>90%)
- Used to detect when the GPU is saturated
- GPU utilization range above threshold (>25%)
- Used to detect uneven distribution of GPU compute workload
- GPU memory utilization below threshold (<10%)
- Used to detect workloads that do not make full use of the GPU or are allocated to an oversized GPU
- GPU memory utilization above threshold (>95%)
- Used to detect when a job is about to run out of GPU memory
- InfiniBand receive/transmit > 0 when running multi-node workloads
- Used to identify workloads that are not properly configured to use InfiniBand
ChatGPT 4o
#
If the behavior is chronic and unproductive, decide how much engagement is worthwhile.
Sometimes planting a seed is better than trying to change their mind in the moment.
Some people are resistant to new perspectives. If they refuse to engage, focus on managing your own reaction rather than changing theirs.
Choose your battles: Not every conversation is worth having. Consider whether it’s worth investing time and energy into trying to change someone’s mind.
ChatGPT OSS 20B
#
Change is a marathon, not a sprint; pace yourself and your words.
Every conversation is a decision.
Here is the list of python tools and libraries I use regularly.
(Sorted alphabetically)
Reference
#
In this article I list all the AI tools I’ve used.
I also try to keep this list up to date with regards to whether I’m still using the tool or not.
Legend:
- 🟢 Regularly using
- 🟡 Sometimes using
- 🔴 Not using
Text generation
#
Code generation
#
Image generation
#
Audio generation
#
Task orchestration
#
Speech to text
#
Desktop client
#
Web client
#
Models
#
- 🔴 Composer-1
- 🔴 Composer-1.5
- 🔴 Composer-2
- 🔴 Xomposer-2.5
- 🔴 Claude Fable 5
- 🔴 Claude Opus 4.1
- 🔴 Claude Opus 4.5
- 🔴 Claude Opus 4.6
- 🔴 Claude Opus 4.7
- 🔴 Claude Opus 4.8
- 🔴 Claude Opus 5
- 🔴 Claude Sonnet 4.5
- 🔴 Claude Sonnet 5
- 🔴 DeepSeek r1
- 🟢 DeepSeek v4 Flash
- 🔴 Gemini 2.5 Pro
- 🔴 Gemini 3 Pro
- 🔴 GLM 4.6
- 🔴 GLM 4.7
- 🔴 GLM 5
- 🔴 GLM 5.1
- 🔴 GLM 5.2
- 🟢 GLM 5.3
- 🟢 GLM 5.3 Flash
- 🔴 GPT 3.5
- 🔴 GPT 4
- 🔴 GPT 4.1
- 🔴 GPT 4o
- 🔴 GPT 5
- 🔴 GPT 5.1
- 🔴 GPT 5.2
- 🔴 GPT 5.2
- 🔴 GPT 5.3
- 🔴 GPT 5.4
- 🔴 GPT 5.6 Sol
- 🔴 GPT OSS 120B
- 🔴 GPT OSS 20B
In this article we’ll cover adding rate limiting to an Apache Flink pipeline. While Apache Flink already contains some APIs that implement some form of rate limiting indirectly, such as AsyncDataStream through a capacity limit which limits the number of concurrent executions, these do not rate limit directly.
Rate limiting is critical in applications where calling 3rd party APIs with rate limits or quotas could result in multiple unnecessary retries or even failures to process an event successfully.
- Implementation of rate limiting
- This consist mostly in including a call to a blocking implementation of a rate limiter just before you would execute the code you want rate limited. For example if you are rate limiting calls to an API, you would block just before sending your HTTP request.
- Rate limiting in relation to parallelism
- When defining rate limiting we generally think of the overall rate limit per second. With Apache Flink parallelism, this number ends up being used by each instance, which makes the rate limit useless in this situation, unless it is divided by the number of parallel instances running the code that is rate limited.
- One way to implement this local rate limit is given in the
open function in 
Flink GuavaFlinkConnectorRateLimiter by dividing the global rate limit by runtimeContext.getTaskInfo().getNumberOfParallelSubtasks()
- In a scenario where we could increase the parallelism indefinitely, what will determine if we autoscale will be how busy the vertex containing our rate limiting operation is. If we have a rate limit of 100/s, it will be 100% busy as soon as we’re processing that many and are rate limited.
- In most situations however we will want to cap our total rate so we will need to configure a maximum parallelism.
- I’ve always thought that we should be making use of the full rate limit we’re allowed, such that even with a parallelism of 1, if our rate limit is 1000/s, that is our rate limit. If the vertex containing the rate limited operation scales to 2, the rate limit of each operator would now be 500.
- Earlier we mentioned that busyness was used to determine when to scale. In this scenario, it seems we would only scale once we reached 1000/s. But then scaling would not help us, as we would simply be 100% busy but on more instances.
- This approach only makes sense if scaling happens for a different reason than rate limiting, such as getting capacity limited or because we cannot reach the rate limit on a single instance. Another reason could be that the rest of the operations in the vertex make it highly busy.
- The downside of specifying a rate limit that is smaller than the total rate limit you have is that you will need to scale possibly unnecessarily because the instance could have handled all the rate limited operations of the global limit.
- What options do we have?
- Keep the rate limited in the same vertex and divide the global rate limit by the number of parallel subtasks
- This allows us to benefit from lower overhead communication between operators
- Isolate the rate limited operation in its own chain
- This allows the operation to scale independently of what happens before and after.
- The downside is increased overhead to transfer the data between the chains before and after.
- Partially isolate the operation in a new chain (either with the operators before or after)
- It’s a trade-off of the benefits of full isolation with no isolation, namely that overhead is partially reduced, but you may affect the parallelism of other operations within the same vertex.